job.setoutputkeyclass和setoutputvalueclass与reducer的context.write方法不匹配,程序仍在正常运行如何处理?

zour9fqk  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(375)

驱动程序代码:

public class WcDriver {

    public static void main(String[] args) throws IOException,
        InterruptedException, ClassNotFoundException {
        Configuration conf = new Configuration();

        Job job = new Job(conf, "WcDriver");
        job.setJarByClass(WcDriver.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        job.setMapperClass(WcMapper.class);
        job.setReducerClass(WcReducer.class);

        job.waitForCompletion(true);
    }
}

减速机代码

public class WcReducer extends Reducer<Text, LongWritable, Text,String>
{   
    @Override
    public void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
        String key1 = null;
        int total = 0;
        for (LongWritable value : values) {
            total += value.get();
            key1= key.toString();
        }
        context.write(new Text(key1), "ABC");
    }
}

在这里,在驾驶课上我设置了 job.setOutputKeyClass(Text.class) 以及 job.setOutputValueClass(LongWritable.class) ,但在减速机课上我在写一个字符串 context.write(new Text(key1), "ABC"); . 我认为在运行程序时应该有一个错误,因为输出类型不匹配,而且reducer的键应该实现 WritableComparable 价值观应该实现 Writable 接口。奇怪的是,这个程序运行得很好。我不明白为什么没有例外。

jdgnovmf

jdgnovmf1#

试试这个

//job.setOutputValueClass(LongWritable.class); if you comment this line you get an error
this will for only define the key value pair by defaul it depent on the output format and
it will be text so this is not giving any error
p1tboqfb

p1tboqfb2#

尝试这样做:

//  job.setOutputFormatClass(TextOutputFormat.class); 
// comment this line, and you'll sure get exception of casting.

这是因为,textoutputformat假设longwritable作为键,text作为值,如果您不定义outputformat类,它将期望获得writable的默认行为,这是默认的,但是如果您提到它,它将隐式地将其强制转换为给定的类型。;

相关问题