我试图理解mapreduce如何对map输出键进行排序,以及它使用的排序算法是什么。我有一个这样的文本文件
a b e f c b
它如何使用这些键执行排序。我实现了一个自定义的sortcomparator类,扩展了writablecomparator接口。我想看看排序是如何进行的,所以我将这些操作写入一个文件中。
public static class MySortComparator2 extends WritableComparator{
@Override
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
Text x=new Text("");
Text y=new Text("");
try {
x.readFields(new DataInputStream(new ByteArrayInputStream(b1)));
y.readFields(new DataInputStream(new ByteArrayInputStream(b2)));
FileWriter writerOfComparisions=new FileWriter("/home/srimanth/Comparisons",true);
writerOfComparisions.write("Comparing "+x.toString()+" with "+y.toString());
writerOfComparisions.write(" ----> returning "+x.compareTo(y));
writerOfComparisions.write("\n");
writerOfComparisions.close();
} catch (IOException e) {
e.printStackTrace();
}
return x.compareTo(y);
}
比较文件的输出如下
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing b with a ----> returning 1
Comparing c with a ----> returning 2
Comparing f with a ----> returning 5
Comparing e with a ----> returning 4
Comparing b with a ----> returning 1
Comparing b with c ----> returning -1
Comparing c with f ----> returning -3
Comparing f with e ----> returning 1
Comparing e with b ----> returning 3
Comparing b with a ----> returning 1
这个排序算法是什么?我的wordcount的最终输出是
b 1
c 1
f 1
e 1
b 1
a 1
看起来mapreduce没有正常运行。这是我的mapper和reducer类。
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
我做错了什么。我会很感激你的帮助。提前谢谢。
1条答案
按热度按时间2jcobegt1#
您是否设置了以下驱动程序代码?
job.setsortcomparatorclass(mysortcomparator2.class);