在运行作业期间,在同一个jar但不同的包中找不到hadoop类

9cbw7uwe  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(304)

我遵循的是《hadoop:权威指南2/e》一书的例子。我遇到一个问题……:(。
我用的是Ubuntu12.04,Hadoop2.2.0。我使用eclipse制作job.jar。类Map\u reduce.programming.v1.maxtemperaturereducer在jar中,但不同的包。当我运行作业时,遇到类未找到异常。
下面是mapred-site.xml

<configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
</configuration>

如果将该值更改为local not yarn,它就起作用了。但如果是Yarn,就不起作用了。hadoop\u类\u路径包括包含job.jar的路径。根本原因是什么?

package map_reduce.programming.v3;

import map_reduce.programming.v1.MaxTemperatureReducer;

import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

/**
 * hadoop map_reduce.programming.v3.MaxTemperatureDriver -conf conf/hadoop-local.xml /book/input/ncdc/micro max-temp
 * hadoop jar job.jar map_reduce.programming.v3.MaxTemperatureDriver -conf conf/hadoop-cluster.xml /book/input/ncdc/all max-temp
 * 
 *
 */
public class MaxTemperatureDriver extends Configured implements Tool {

    @Override
    public int run(String[] args) throws Exception {

        if (args.length != 2) {
            System.err.printf("Usage: %s [generic optins] <input> <output>\n", getClass().getSimpleName());
            ToolRunner.printGenericCommandUsage(System.err);
            return -1;
        }   

        JobConf conf = new JobConf(getConf(), getClass());
        conf.setJobName("Max temperature");     

        FileInputFormat.addInputPath(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);

        conf.setMapperClass(MaxTemperatureMapper.class);
        conf.setCombinerClass(MaxTemperatureReducer.class);
        conf.setReducerClass(MaxTemperatureReducer.class);

        JobClient.runJob(conf);
        return 0;
    }

    public static void main(String[] args) throws Exception {
        int exitCode = ToolRunner.run(new MaxTemperatureDriver(), args);
        System.exit(exitCode);
    }

}

下面是日志。。
jar job.jar map\u reduce.programming.v3.maxtemperaturedriver-conf conf/hadoop-cluster.xml/book/input/ncdc/all max-temp0991

14/06/05 18:10:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/05 18:10:20 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/06/05 18:10:20 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/06/05 18:10:20 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
14/06/05 18:10:20 INFO mapred.FileInputFormat: Total input paths to process : 2
14/06/05 18:10:21 INFO mapreduce.JobSubmitter: number of splits:2
14/06/05 18:10:21 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/06/05 18:10:21 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/06/05 18:10:21 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/06/05 18:10:21 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/06/05 18:10:21 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/06/05 18:10:21 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/06/05 18:10:21 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/06/05 18:10:21 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/06/05 18:10:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1401958773644_0002
14/06/05 18:10:22 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
14/06/05 18:10:22 INFO impl.YarnClientImpl: Submitted application 
application_1401958773644_0002 to ResourceManager at /0.0.0.0:8032
14/06/05 18:10:22 INFO mapreduce.Job: The url to track the job: http://ubuntu:8088/proxy/application_1401958773644_0002/
14/06/05 18:10:22 INFO mapreduce.Job: Running job: job_1401958773644_0002
14/06/05 18:10:27 INFO mapreduce.Job: Job job_1401958773644_0002 running in uber mode : false
14/06/05 18:10:27 INFO mapreduce.Job:  map 0% reduce 0%
14/06/05 18:10:30 INFO mapreduce.Job: Task Id : attempt_1401958773644_0002_m_000001_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class map_reduce.programming.v1.MaxTemperatureReducer not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
    at org.apache.hadoop.mapred.JobConf.getCombinerClass(JobConf.java:1139)
    at org.apache.hadoop.mapred.Task$CombinerRunner.create(Task.java:1517)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1010)
    at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:390)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
3zwjbxry

3zwjbxry1#

我遇到了完全相同的问题,作业配置没有找到Map器和reducer类,并且抛出了classnotfoundexception。
我用的是mapreduce2所以我不得不加上

job.setJarByClass(Avg.class);

如果是你的话,我想你应该打电话

JobConf conf = new JobConf(getConf(), YouClassMain.class);

最好的,爱德华多

相关问题