我希望使用parquet mr库以parquet文件格式编写mapreduce输出,如下所示:
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(ParquetOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[1]));
ParquetOutputFormat.setOutputPath(job, new Path(args[2]));
ParquetOutputFormat.setCompression(job, CompressionCodecName.GZIP);
SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);
SkipBadRecords.setAttemptsToStartSkipping(conf, 0);
job.submit();
但是,我经常会遇到这样的错误:
2018-02-23 09:32:58,325 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException: writeSupportClass should not be null
at org.apache.parquet.Preconditions.checkNotNull(Preconditions.java:38)
at org.apache.parquet.hadoop.ParquetOutputFormat.getWriteSupport(ParquetOutputFormat.java:350)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:293)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:283)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:548)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:622)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
我知道writesupportclass需要像这样传递/设置
ParquetOutputFormat.setWriteSupportClass(job, ProtoWriteSupport.class);
但是我能问一下如何指定schema,实现protowritesupport或者其他writesupport类吗?我需要实现什么方法?有没有正确实现的例子?
如果有帮助的话,我的mr job的输出应该看起来像&以Parquet格式存储:
Text INTWRITABLE
a 100
1条答案
按热度按时间p5fdfcr11#
尝试
ParquetOutputFormat.setWriteSupportClass(job, ProtoWriteSupport.class);
```ProtoWriteSupport