java在我的mapreduce代码中捕获了classcastexception

c2e8gylq  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(387)

这里附上了我的代码,它可以把所有的avro文件混成一个大文件。当我运行它时,classcastexception被捕获,有人能帮我吗?

public class FileCompactionDriver extends Configured implements Tool {

    public static void main(String[] args) throws Exception {

        int returnCode = ToolRunner.run(new FileCompactionDriver(), args);
        /* Terminate job */
        System.exit(returnCode);
    }

    public int run(String[] arg) throws Exception {

        /* Setup configuration */
        Configuration conf = this.getConf();

        /* Test if it is a valid running command */
        if (arg.length != 2) {
            System.err.println("Usage: FileCompaction <input path> <output path>");
            System.exit(2);
        }

        // Creating the job object from configuration
        Job job = Job.getInstance(conf);

        job.setJobName("FileCompactor");
        job.setJarByClass(FileCompactionMapper.class);

        /* Setup the Mapper and Reducer classes */
        job.setMapperClass(FileCompactionMapper.class);
        job.setReducerClass(FileCompactionReducer.class);
        job.setNumReduceTasks(1);

        /* Setup the output types */
        job.setOutputKeyClass(AvroKey.class);
        job.setOutputValueClass(NullWritable.class);

        /* Setup the input and output classes format */
        /* Handle and writing avro container files */
        job.setInputFormatClass(AvroKeyInputFormat.class);
        job.setOutputFormatClass(AvroKeyOutputFormat.class);

        /* Specify input and output directories */
        FileInputFormat.setInputPaths(job, new Path(arg[0]));
        FileOutputFormat.setOutputPath(job, new Path(arg[1]));

        return (job.waitForCompletion(true) ? 0 : 1);
    }
}
nukf8bse

nukf8bse1#

我想你需要具体说明一下 job.setMapOutputKeyClass 以及 job.setMapOutputValueClass 摆脱 ClassCastException .

相关问题