这里附上了我的代码,它可以把所有的avro文件混成一个大文件。当我运行它时,classcastexception被捕获,有人能帮我吗?
public class FileCompactionDriver extends Configured implements Tool {
public static void main(String[] args) throws Exception {
int returnCode = ToolRunner.run(new FileCompactionDriver(), args);
/* Terminate job */
System.exit(returnCode);
}
public int run(String[] arg) throws Exception {
/* Setup configuration */
Configuration conf = this.getConf();
/* Test if it is a valid running command */
if (arg.length != 2) {
System.err.println("Usage: FileCompaction <input path> <output path>");
System.exit(2);
}
// Creating the job object from configuration
Job job = Job.getInstance(conf);
job.setJobName("FileCompactor");
job.setJarByClass(FileCompactionMapper.class);
/* Setup the Mapper and Reducer classes */
job.setMapperClass(FileCompactionMapper.class);
job.setReducerClass(FileCompactionReducer.class);
job.setNumReduceTasks(1);
/* Setup the output types */
job.setOutputKeyClass(AvroKey.class);
job.setOutputValueClass(NullWritable.class);
/* Setup the input and output classes format */
/* Handle and writing avro container files */
job.setInputFormatClass(AvroKeyInputFormat.class);
job.setOutputFormatClass(AvroKeyOutputFormat.class);
/* Specify input and output directories */
FileInputFormat.setInputPaths(job, new Path(arg[0]));
FileOutputFormat.setOutputPath(job, new Path(arg[1]));
return (job.waitForCompletion(true) ? 0 : 1);
}
}
1条答案
按热度按时间nukf8bse1#
我想你需要具体说明一下
job.setMapOutputKeyClass
以及job.setMapOutputValueClass
摆脱ClassCastException
.