hadoop top10树Map没有输出任何数据

irlmq6kh  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(273)

我有java代码,应该Map一个top10树Map hadoop . 然而,当我运行代码时,我没有得到任何数据输出。我哪里漏了什么?
代码:

public class Mapper1_1 extends Mapper<Object, Text, NullWritable, Text> {

 public static final int limit = 10;
 private TreeMap<Integer, Text> top10cats = new TreeMap<Integer, Text>();
 private int flag=0;
 private int tmp=0;

@Override
protected void map(Object key, Text value,
        Mapper<Object, Text, NullWritable, Text>.Context context)
        throws IOException, InterruptedException {
    String line=value.toString();
    if(!line.isEmpty()){
    String[] words=line.split("\t");
    if(flag!=5){
         System.out.println(words[0]+"&&&&&&&&&&&&"+words[1]);
         flag++;
    }
    top10cats.put(Integer.parseInt(words[1]), value);
    if(top10cats.size()>limit){
        tmp++;
        top10cats.remove(top10cats.firstKey());
    }
    }
}

@Override
protected void cleanup(
        Mapper<Object, Text, NullWritable, Text>.Context context)
        throws IOException, InterruptedException {
    System.out.println("&&&&&&&"+tmp);
     for(Text text: top10cats.values()){
         System.out.println("##########"+text.toString());
         context.write(NullWritable.get(), text);
        }
}
}

public class Reducer1_1 extends Reducer<NullWritable, Text, NullWritable, Text> {

public static final int N = 10;
private TreeMap<Integer, Text> top10cats = new TreeMap<Integer, Text>();

@Override
protected void reduce(NullWritable key, Iterable<Text> values,
        Reducer<NullWritable, Text, NullWritable, Text>.Context context)
        throws IOException, InterruptedException {
    for (Text val : values) {
             String tokens[] = val.toString().split("\t");
             Integer weight = Integer.parseInt(tokens[1]);
             top10cats.put(weight, val);
             if (top10cats.size() > N)
                 top10cats.remove(top10cats.firstKey());
         }
    for(Text text:top10cats.values()){
        context.write(NullWritable.get(), text);
    }
}
}

运行程序时的结果输出:

2018-07-10 09:16:19,220 INFO [org.apache.hadoop.mapred.MapTask] -
   mapreduce.task.io.sort.mb: 100 2018-07-10 09:16:19,220 INFO
   [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
   2018-07-10 09:16:19,220 INFO [org.apache.hadoop.mapred.MapTask] -
   bufstart = 0; bufvoid = 104857600 2018-07-10 09:16:19,220 INFO
   [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length =
   6553600 2018-07-10 09:16:19,226 INFO
   [org.apache.hadoop.mapred.MapTask] - Map output collector class =
   org.apache.hadoop.mapred.MapTask$MapOutputBuffer
   1000473355&&&&&&&&&&&&39 
   1000473355   39 
   1001103582&&&&&&&&&&&&6
   1001103582   6 
   1001698581&&&&&&&&&&&&2 
   1001698581   2
   1004307115&&&&&&&&&&&&24 
   1004307115   24 
   1005179492&&&&&&&&&&&&2
   1005179492   2 
   &&&&&&&1605
   ##########
   ##########
   ##########
   ##########
   ##########
   ##########
   ##########
   ##########
   ##########
   ##########

2018-07-10 09:16:19287信息[org.apache.hadoop.mapred.localjobrunner]-2018-07-10 09:16:19287信息[org.apache.hadoop.mapred.maptask]-开始刷新Map输出2018-07-10 09:16:19288信息[org.apache.hadoop.mapred.maptask]-溢出Map输出2018-07-10 09:16:19288信息[org.apache.hadoop.mapred.maptask]-bufstart=0;bufend=10;bufvoid=104857600 2018-07-10 09:16:19291信息[org.apache.hadoop.mapred.maptask]-kvstart=26214396(104857584);kvend=26214360(104857440);length=37/6553600 2018-07-10 09:16:19307 info[org.apache.hadoop.mapred.maptask]-开始刷新Map输出2018-07-10 09:16:19307 info[org.apache.hadoop.mapred.maptask]-(重置)赤道0 kv 26214396(104857584)kvi 26214356(104857424)2018-07-10 09:16:19,307 info[org.apache.hadoop.mapred.maptask]-spilling map output 2018-07-10 09:16:19307 info[org.apache.hadoop.mapred.maptask]-bufstart=0;bufend=10;bufvoid=104857600 2018-07-10 09:16:19727信息[org.apache.hadoop.mapred.maptask]-kvstart=26214396(104857584);kvend=26214360(104857440);length=37/6553600 2018-07-10 09:16:19744 info[org.apache.hadoop.mapred.maptask]-关闭org.apache.hadoop.mapred时忽略异常。maptask$newoutputcollector@61ccfc89 java.lang.arrayindexoutofboundsexception:1位于com.zy.top10.reducer1\u 1.reduce(reducer1\u 1。java:21)在com.zy.top10.reducer1\u 1.reduce(reducer1\u 1。java:1)
问题:system.out.println(“###########”+text.tostring());text.tostring()为空。为什么?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题