在hadoop mr任务中使用avro作为Map器的输出格式时,如何设置压缩编解码器?
旧的“mapred”api提供了以下方法:
org.apache.avro.mapred.AvroJob.setOutputCodec(JobConf job, String codec)
但是,这在较新的“mapreduce”api中是不存在的。如何在较新的“mapreduce”api中设置编解码器?
我天真地尝试使用作业配置设置编解码器,但没有成功:
public int run(String[] args) throws Exception {
[..]
Job job = new Job(getConf());
job.setJarByClass(MapReduceExample.class);
job.setJobName("MRExample");
// hm .. this doesn't seem to do work, output still has "null" codec
job.getConfiguration().set(AvroJob.CONF_OUTPUT_CODEC,
CodecFactory.deflateCodec(6).toString());
job.setMapperClass(ExampleMapper.class);
[..]
AvroJob.setMapOutputKeySchema(job, Schema.create(Schema.Type.STRING));
AvroJob.setMapOutputValueSchema(job, Schema.create(Schema.Type.BYTES));
// here I was hoping to use something like
// AvroJob.setMapOutputCodec(job, "deflate")
[..]
return (job.waitForCompletion(true) ? 0 : 1);
}
当我用python打开生成的avro时
>>> from avro.datafile import DataFileReader
>>> from avro.io import DatumReader
>>> av_fh = open("output/part-r-00000.avro", "rb")
>>> av_rd = DataFileReader(av_fh, DatumReader())
>>> av_rd.codec
'null'
1条答案
按热度按时间ukxgm1gy1#
当我更改以下行时,它就起作用了
到