Map程序无法读取gz.parquet文件

1aaf6o9v  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(368)

org.apache.hadoop.mapred.maptask:开始刷新Map输出
2017-04-20 20:53:20,101 warn[main]org.apache.hadoop.mapred.yarnchild:运行子项的异常:java.lang.nullpointerexception位于org.apache.parquet.avro.avroschemaconverter.convertfield(avroschemaconverter)。java:294)在org.apache.parquet.avro.avroschemaconverter.convertfields(avroschemaconverter。java:204)在org.apache.parquet.avro.avroschemaconverter.convert(avroschemaconverter。java:198)在org.apache.parquet.avro.avroreadsupport.prepareforead(avroreadsupport。java:105)位于org.apache.parquet.hadoop.internalparquetrecordreader.initialize(internalparquetrecordreader)。java:174)在org.apache.parquet.hadoop.parquetrecordreader.initializeinternalreader(parquetrecordreader。java:192)在org.apache.parquet.hadoop.parquetrecordreader.initialize(parquetrecordreader。java:140)位于org.apache.hadoop.mapreduce.lib.input.delegatingrecordreader.initialize(delegatingrecordreader)。java:84)在org.apache.hadoop.mapred.maptask$newtrackingrecordreader.initialize(maptask。java:548)在org.apache.hadoop.mapred.maptask.runnewmapper(maptask。java:786)在org.apache.hadoop.mapred.maptask.run(maptask。java:341)在org.apache.hadoop.mapred.yarnchild$2.run(yarnchild。java:163)位于java.security.accesscontroller.doprivileged(本机方法)javax.security.auth.subject.doas(主题。java:415)在org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation。java:1657)在org.apache.hadoop.mapred.yarnchild.main(yarnchild。java:158)
我尝试在mapper中读取filename.gz.paquet,使用相同的mapreduce作业,我可以读取filename.snappy.parquet文件。

8dtrkrch

8dtrkrch1#

我可以解决这个问题。嗯,我已经更新了avroversion1.8.1、parquetversion1.9.0、parquetformatversion2.3.1和hiveversion1.2.2的新jar。

相关问题