如何设置Map器输出的avro压缩编解码器?

mccptt67  于 2021-05-31  发布在  Hadoop
关注(0)|答案(1)|浏览(390)

在hadoop mr任务中使用avro作为Map器的输出格式时,如何设置压缩编解码器?
旧的“mapred”api提供了以下方法:

org.apache.avro.mapred.AvroJob.setOutputCodec(JobConf job, String codec)

但是,这在较新的“mapreduce”api中是不存在的。如何在较新的“mapreduce”api中设置编解码器?
我天真地尝试使用作业配置设置编解码器,但没有成功:

public int run(String[] args) throws Exception {
  [..]
  Job job = new Job(getConf());
  job.setJarByClass(MapReduceExample.class);
  job.setJobName("MRExample");
  // hm .. this doesn't seem to do work, output still has "null" codec
  job.getConfiguration().set(AvroJob.CONF_OUTPUT_CODEC,
                             CodecFactory.deflateCodec(6).toString());

  job.setMapperClass(ExampleMapper.class);
  [..]
  AvroJob.setMapOutputKeySchema(job, Schema.create(Schema.Type.STRING));
  AvroJob.setMapOutputValueSchema(job, Schema.create(Schema.Type.BYTES));
  // here I was hoping to use something like
  // AvroJob.setMapOutputCodec(job, "deflate")

  [..]
  return (job.waitForCompletion(true) ? 0 : 1);
}

当我用python打开生成的avro时

>>> from avro.datafile import DataFileReader
>>> from avro.io import DatumReader
>>> av_fh = open("output/part-r-00000.avro", "rb")
>>> av_rd = DataFileReader(av_fh, DatumReader())
>>> av_rd.codec
'null'
ukxgm1gy

ukxgm1gy1#

当我更改以下行时,它就起作用了

job.getConfiguration().set(AvroJob.CONF_OUTPUT_CODEC,
                           CodecFactory.deflateCodec(6).toString());

FileOutputFormat.setCompressOutput(job, true);
job.getConfiguration().set(AvroJob.CONF_OUTPUT_CODEC,
                           DataFileConstants.DEFLATE_CODEC);

相关问题