在HDP2.0上运行hadoop map reduce java程序时出错

tzxcd3kk  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(334)

我在java中运行mapreduce程序,它编译成功,但在运行时我在ec2上使用hortonworks hdp2.0。

package edu.osu.cse.ysmart.TestQuery;
    import java.io.IOException;
    import java.util.*;
    import java.text.*;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.conf.*;
    import org.apache.hadoop.io.*;
    import org.apache.hadoop.util.Tool;
    import org.apache.hadoop.util.ToolRunner;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.input.FileSplit;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
    import org.apache.hadoop.mapreduce.lib.partition.*;

    public class TestQuery1 extends Configured implements Tool{

            public static class Map extends Mapper<Object, Text,NullWritable,Text>{

                    public void map(Object key, Text value, Context context) throws IOException,InterruptedException{

                            String line = value.toString();
                            String[] line_buf = new String[3];
                            int prev=0,i=0,n=0;
                            for(i=0,n=0,prev=0;i<line.length();i++){

                                    if (line.charAt(i) == '|'){
                                            line_buf[n] = line.substring(prev,i);
                                            n = n+1;
                                            prev = i+1;
                                    }
                                    if(n == 3)
                                            break;
                            }

                            if(n<3)
                                    line_buf[n] = line.substring(prev,i);
                            if(line_buf[2].compareTo("Abid Khan") == 0){

                                    NullWritable key_op = NullWritable.get();
                                    context.write(key_op , new Text(line_buf[1]+ "|" ));
                            }
                    }

            }

            public int run(String[] args) throws Exception{

                    Configuration conf = new Configuration();
                    Job job = new Job(conf,"TestQuery1");
                    job.setJarByClass(TestQuery1.class);
                    job.setMapOutputKeyClass(NullWritable.class);
                    job.setMapOutputValueClass(Text.class);
                    job.setOutputKeyClass(NullWritable.class);
                    job.setOutputValueClass(Text.class);
                    job.setMapperClass(Map.class);
                    FileInputFormat.addInputPath(job, new Path(args[0]));
                    FileOutputFormat.setOutputPath(job, new Path(args[1]));
                    return (job.waitForCompletion(true) ? 0 : 1);
            }

            public static void main(String[] args) throws Exception {

                            int res = ToolRunner.run(new Configuration(), new TestQuery1(), args);
                            System.exit(res);
            }

    }

下面是错误的原因。。。
hadoop jar/usr/local/read.jar edu.osu.cse.ysmart.testquery.testquery1/user/nitin//resume.csv/user/nitin/outs 14/01/03 17:12:05 info client.rmproxy:连接到resourcemanager的ip-10-0-0-219.ec2.internal/10.0.0.219:8050 14/01/03 17:12:07 warn mapreduce.jobsubmitter:未执行hadoop命令行选项解析。实现工具接口并使用toolrunner执行应用程序以解决此问题。14/01/03 17:12:07 info input.fileinputformat:要处理的总输入路径:1 14/01/03 17:12:07 info mapreduce.jobsubmitter:数量splits:1 14/01/03 17:12:07 info configuration.deprecation:user.name已弃用。相反,请使用mapreduce.job.user.name 14/01/03 17:12:07 info configuration.deprecation:mapred.jar。相反,请使用mapreduce.job.jar 14/01/03 17:12:07 info configuration.deprecation:mapred.output.value.class已弃用。相反,请使用mapreduce.job.output.value.class 14/01/03 17:12:07 info configuration.deprecation:mapred.mapoutput.value.class已弃用。相反,请使用mapreduce.map.output.value.class 14/01/03 17:12:07 info configuration.deprecation:mapreduce.map.class已弃用。相反,请使用mapreduce.job.map.class 14/01/03 17:12:07 info configuration.deprecation:mapred.job.name已弃用。相反,请使用mapreduce.job.name 14/01/03 17:12:07 info configuration.deprecation:mapred.input.dir。相反,请使用mapreduce.input.fileinputformat.inputdir 14/01/03 17:12:07 info configuration.deprecation:mapred.output.dir已弃用。相反,请使用mapreduce.output.fileoutputformat.outputdir 14/01/03 17:12:07 info configuration.deprecation:mapred.map.tasks。相反,请使用mapreduce.job.maps 14/01/03 17:12:07 info configuration.deprecation:mapred.output.key.class。相反,请使用mapreduce.job.output.key.class 14/01/03 17:12:07 info configuration.deprecation:mapred.mapoutput.key.class已弃用。相反,请使用mapreduce.map.output.key.class 14/01/03 17:12:07 info configuration.deprecation:mapred.working.dir。相反,使用mapreduce.job.working.dir 14/01/03 17:12:08 info mapreduce.jobsubmitter:为作业提交令牌:job\u 1388768203\u 0006 14/01/03 17:12:08 info impl.yarclientimpl:已将应用程序应用程序\u 1388768204303\u 0006提交到resourcemanager,地址为ip-10-0-0-219.ec2.内部/10.0.0.219:8050 14/01/03 17:12:08 info mapreduce.job:跟踪工作:http://ip.ec2.internal:8088/proxy/application\u 138876820483\u 0006/14/01/03 17:12:08 info mapreduce.job:正在运行作业:job\u 138876820483\u 0006 14/01/03 17:12:36 info mapreduce.job:正在uber模式下运行的作业:false 14/01/03 17:12:36 info mapreduce.job:map 0%reduce 0%14/01/03 17:12:58 info mapreduce.job:任务id:尝试\u 1388768204803 \u 0006 \u m \u 0000000,状态:失败错误:java.lang.nullpointerexception at edu.osu.cse.ysmart.testquery.testquery1$map.map(testquery1)。java:42)在edu.osu.cse.ysmart.testquery.testquery1$map.map(testquery1。java:22)在org.apache.hadoop.mapreduce.mapper.run(mapper。java:145)在org.apache.hadoop.mapred.maptask.runnewmapper(maptask。java:763)在org.apache.hadoop.mapred.maptask.run(maptask。java:339)在org.apache.hadoop.mapred.yarnchild$2.run(yarnchild。java:162)位于javax.security.auth.subject.doas(subject)的java.security.accesscontroller.doprivileged(本机方法)。java:396)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1491)在org.apache.hadoop.mapred.yarnchild.main(yarnchild。java:157)
请帮忙

lb3vh1jj

lb3vh1jj1#

从异常跟踪中可以看出,您试图调用空对象上的方法:

attempt_1388768204803_0006_m_000000_0, Status : FAILED Error: java.lang.NullPointerException at edu.osu.cse.ysmart.TestQuery.TestQuery1$Map.map(TestQuery1.java:42) at

从源代码看,以下比较失败:

if(line_buf[2].compareTo("Abid Khan") == 0)

没有填充行\u buf[2],因此出现空指针异常。您可以在if条件之前打印第\u buf[2]行的值,以查看是否有值。这将帮助您调试问题。

相关问题