mapreduce程序Map任务超时

zazmityj  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(319)

我犯了个奇怪的错误。我编写了一个wordcount程序来计算一个单词在文件中重复的次数。
因此,当我在hadoop上运行mr代码时,代码会停留在“map 100%,reduce 0%”上。基本模式是第一个map任务在600秒后超时,然后再超时一次,作业就会自杀。
我检查了job tracker,任务被卡住了,因为map任务没有完成reduce任务。
我已经试着修复了2天,在这期间我删除了原来的虚拟ubuntucloudera并再次安装了它-所以我们可以确定这不是一个配置问题。
感谢您的帮助。
下面是3个代码文件。
字数.java

public class WordCount extends Configured implements Tool {

@Override
public int run(String[] args) throws Exception {

    Configuration conf =  super.getConf();

    Job job=new Job(conf, "Word Count Job");
    job.setJarByClass(WordCount.class);

    job.setMapperClass(WordMapper.class);
    job.setReducerClass(WordReducer.class);

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(FloatWritable.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    FileInputFormat.setInputPaths(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.waitForCompletion(Boolean.TRUE);

    return 0;
}

public static void main(String[] args) {

    //Display error message in case insufficient arguments supplied
    if(args.length<2){
        System.out.println("usage: WordCount <Input-Path> <Output-Path>");
    }

    Configuration conf=new Configuration(Boolean.TRUE);

    int i;
    try {
        //Run the overridden 'run' method code
        i = ToolRunner.run(conf, new WordCount(), args);

        //Print usage stats to out
        //ToolRunner.printGenericCommandUsage(System.out);

        //exit if job cannot start
        System.exit(i);

    } catch (Exception e) {

        e.printStackTrace();

        System.exit(-1);
    }
}
}

wordmapper.java文件

public class WordMapper extends Mapper<LongWritable, Text, Text, FloatWritable> {

@Override
protected void map(LongWritable key, 
        Text value,
        Mapper<LongWritable, Text, Text, FloatWritable>.Context context)
        throws IOException, InterruptedException {

    if(!value.toString().trim().isEmpty()){

        StringTokenizer valTokens = new StringTokenizer(value.toString()); 

        while(valTokens.hasMoreTokens()){
            context.write(new Text(valTokens.nextToken()), new FloatWritable(Float.parseFloat("1.00")));
        }
    }   
}
}

wordreducer.java文件

public class WordReducer extends Reducer<Text, FloatWritable, Text, FloatWritable> {

@Override
protected void reduce(Text key, Iterable<FloatWritable> values,
        Reducer<Text, FloatWritable, Text, FloatWritable>.Context context)
        throws IOException, InterruptedException {

    Iterator<FloatWritable> valsIter = values.iterator();
    int i = 0;

    while(valsIter.hasNext()) 
        i++;

    context.write(key, new FloatWritable((float)i));
}
}
hjzp0vay

hjzp0vay1#

您的问题在这行代码中:

while(valsIter.hasNext()) 
    i++;

hasnext检查迭代器中是否有下一个元素,但不移动指针的位置。因此,检查总是返回true。除非调用valsiter.next()。

相关问题