我不知道如何使用multipleoutputs类。我用它来创建多个输出文件。下面是我的驱动程序类的代码片段
Configuration conf = new Configuration();
Job job = Job.getInstance(conf);
job.setJarByClass(CustomKeyValueTest.class);//class with mapper and reducer
job.setOutputKeyClass(CustomKey.class);
job.setOutputValueClass(Text.class);
job.setMapOutputKeyClass(CustomKey.class);
job.setMapOutputValueClass(CustomValue.class);
job.setMapperClass(CustomKeyValueTestMapper.class);
job.setReducerClass(CustomKeyValueTestReducer.class);
job.setInputFormatClass(TextInputFormat.class);
Path in = new Path(args[1]);
Path out = new Path(args[2]);
out.getFileSystem(conf).delete(out, true);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
MultipleOutputs.addNamedOutput(job, "islnd" , TextOutputFormat.class, CustomKey.class, Text.class);
LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class);
MultipleOutputs.setCountersEnabled(job, true);
boolean status = job.waitForCompletion(true);
在减速机中,我使用了这样的倍数输出,
private MultipleOutputs<CustomKey, Text> multipleOutputs;
@Override
public void setup(Context context) throws IOException, InterruptedException {
multipleOutputs = new MultipleOutputs<>(context);
}
@Override
public void reduce(CustomKey key, Iterable<CustomValue> values, Context context) throws IOException, InterruptedException {
...
multipleOutputs.write("islnd", key, pop, key.toString());
//context.write(key, pop);
}
public void cleanup() throws IOException, InterruptedException {
multipleOutputs.close();
}
}
当我使用context.write时,我会得到包含数据的输出文件。但是当我删除context.write时,输出文件是空的。但我不想调用context.write,因为它会创建额外的文件part-r-00000。如这里所述(类描述的最后一段),我使用lazyoutputformat来避免part-r-00000文件。但还是没用。
1条答案
按热度按时间tzdcorbm1#
lazyoutputformat.setoutputformatclass(作业,textoutputformat.class);
这意味着,如果您没有创建任何输出,请不要创建空文件。
用于多输出的代码是http://bytepadding.com/big-data/map-reduce/multipleoutputs-in-map-reduce/