基于reducer值的升序排序

5lhxktic 于 2021-06-02 发布在 Hadoop

关注(0)|答案(2)|浏览(414)

我是hadoop mapreduce编程范例的新手，有人能告诉我如何根据值轻松排序吗？我尝试实现另一个comparator类，但是有没有一种更简单的方法，比如通过job config根据reducer的值进行排序。基本上，我正在阅读日志文件，我想按升序排序的命中率的网址。

public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {

private final static IntWritable ONE = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
                ) throws IOException, InterruptedException {
    String[] split = value.toString().split(" ");
    for(int i=0; i<split.length; i++){
        if (i==6)
            word.set(split[i]);
            context.write(word, ONE);
    }
}
}

public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> {       
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,  Context context) throws IOException, InterruptedException {
  int sum = 0;
  for (IntWritable val : values) {
    sum += val.get();
  }
  result.set(sum);
  context.write(key, result);      
}
}

Java hadoop mapreduce hadoop2

来源：https://stackoverflow.com/questions/47128975/ascending-sort-based-on-values-of-the-reducer

2条答案

按热度按时间

k5hmc34c1#

在这种情况下，您必须编写两个map reduce作业。第一个作业计算URL。就像第一个作业的输出一样-

yahoo.com,100
google.com,200 
msn.com,50

将此传递给第二个map reduce作业并根据计数对其排序。

赞(0）回复(0）举报 2021-06-02

gcmastyq2#

在reducer类中声明一个map，并将键和值放在map中。现在在reducer类的cleanup（）方法中，尝试按值对Map进行排序，最后在context.write（key，value）中给出值；

public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> {       
private IntWritable result = new IntWritable();

TreeMap<Text,IntWritable>result=new TreeMap<Text, IntWritable>();

public void reduce(Text key, Iterable<IntWritable> values,  Context context) throws IOException, InterruptedException {
  int sum = 0;
  for (IntWritable val : values) {
    sum += val.get();
  }
 result.put(new Text(key),new IntWritable(sum));
}
}

    @Override
    protected void cleanup(Context context)
            throws IOException, InterruptedException {

        Set<Entry<Text, IntWritable>> set = result.entrySet();
        List<Entry<Text, IntWritable>> list = new ArrayList<Entry<Text,IntWritable>>(set);
        Collections.sort( list, new Comparator<Map.Entry<Text, IntWritable>>()
        {
            public int compare( Map.Entry<Text, IntWritable> o1, Map.Entry<Text,IntWritable> o2 )
            {
                return (o2.getValue()).compareTo( o1.getValue() );
            }
        });
        for(Map.Entry<Text,IntWritable> entry:list){

            context.write(entry.getKey(),entry.getValue());
        }

    }
    }

赞(0）回复(0）举报 2021-06-02

我来回答

基于reducer值的升序排序

2条答案

相关问题

热门标签

最新问答