来自spring启动应用程序的hadoop distcp-classnotfoundexception

mwkjh3gx  于 2021-06-01  发布在  Hadoop
关注(0)|答案(1)|浏览(421)

我正在尝试通过restapi调用从spring启动应用程序提交distcp作业。
spring版本:1.5.13.发布hadoop版本:2.7.3
下面是我用来示例化distcp的代码:

List<Path> srcPathList = new ArrayList<Path>();
srcPathList.add(new Path("hdfs://<cluster>/tmp/<user>/source"));

Path targetPath = new Path("hdfs://<cluster>/tmp/<user>/destination");

DistCpOptions distCpOptions = new DistCpOptions(srcPathList,targetPath);
DistCp distCp = new DistCp(configuration,distCpOptions);
Job job = distCp.execute();

作业已成功提交到群集,但由于群集上的classnotfoundexception,作业失败。以下是例外情况:

INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; 
cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:  
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
Class org.apache.hadoop.tools.mapred.CopyOutputFormat not found

为什么会这样?任何关于这方面的建议都会很有帮助!!谢谢!

dxpyg8gm

dxpyg8gm1#

我通过查看 job.jar 在nodemanager机器上。job.jar的结构是:

BOOT-INF/class/xxx

这是不合理的。
我试着用war替换jar包,它成功了!

<packaging>war</packaging>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<!--exclude inner tomcat-->
    <exclusions>
        <exclusion>
            <artifactId>spring-boot-starter-tomcat</artifactId>
            <groupId>org.springframework.boot</groupId>
        </exclusion>
    </exclusions>
</dependency>
<!-- include tomcat-->
<dependency>
    <groupId>org.apache.tomcat</groupId>
    <artifactId>tomcat-servlet-api</artifactId>
    <version>7.0.47</version>
    <scope>provided</scope>
</dependency>
...

然后添加开始类:

import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.support.SpringBootServletInitializer;

public class SpringBootStartApplication extends SpringBootServletInitializer {

    @Override
    protected SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
        // 
        return builder.sources(xxxPortalApplication.class);
    }
}

相关问题