我为mapreduce文本排序编写了这样的代码:
public static class SortMapper extends Mapper<Object, Text, Text, Text> {
private Text citizenship = new Text();
@Override
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
citizenship.set(value.toString().split(",")[11]);
context.write(citizenship, value);
}
}
public static class PrintReducer extends Reducer<Text, Text, NullWritable, Text> {
@Override
protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
Iterator<Text> valIt = values.iterator();
while (valIt.hasNext()) {
Text value = valIt.next();
context.write(NullWritable.get(), value);
}
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "Football Sort");
job.setJarByClass(FootballSort.class);
job.setMapperClass(SortMapper.class);
job.setCombinerClass(PrintReducer.class);
job.setReducerClass(PrintReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(NullWritable.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
但它总是很吸引人
第26、34行出现ioexception原因:org.apache.hadoop.io.nullwriteable类不是org.apache.hadoop.io.text类
2条答案
按热度按时间eivnm1vs1#
@abhinay:在这种情况下不能使用组合器。组合器是操作是可交换和关联的小型还原器,组合器的签名应与还原器匹配。如果组合器签名为“”,则会出现错误,因为还原器的输入键和值为--text和intwritable,但组合器的输出键和值类为text,nullwritable–2015年12月28日5:51
//job.setcombinerclass(printreducer.class);或者删除这个字符串是解决问题的方法
bihw5rsg2#
你的
mapper outputformat
与代码不匹配,在main方法中设置输出文本但在你的Map里
public static class PrintReducer extends Reducer<Text, Text, NullWritable, Text>
你的孩子让他们NullWritable TEXT