我使用的是emr-4.3.0,spark 1.6.0,hive 1.0.0。
我这样写一个表(伪代码)-
val df = <a dataframe>
df.registerTempTable("temptable")
sqlContext.setConf("hive.exec.dynamic.partition.mode","true")
sqlContext.sql("create external table exttable ( some columns ... )
partitioned by (partkey int) stored as parquet location 's3://some.bucket'")
sqlContext.sql("insert overwrite exttable partition(partkey) select columns from
temptable")
写得很好,我可以用-
sqlContext.sql("select * from exttable")
但是,当我尝试使用hive读取表时-
hive -e 'select * from exttable'
配置单元抛出一个nullpointerexception,堆栈跟踪如下。感谢您的帮助-
请求时间=[0.008],响应处理时间=[0.295],httpclientsendrequesttime=[0.313],2016-05-19 03:08:02,537错误[main()]:clidriver(sessionstate。java:printerror(833))-失败,出现异常java.io.ioexception:java.lang.nullpointerexception java.io.ioexception:java.lang.nullpointerexception位于org.apache.hadoop.hive.ql.exec.fetchoperator.getnextrow(fetchoperator)。java:663)在org.apache.hadoop.hive.ql.exec.fetchoperator.pushrow(fetchoperator。java:561)在org.apache.hadoop.hive.ql.exec.fetchtask.fetch(fetchtask。java:138)在org.apache.hadoop.hive.ql.driver.getresults(driver。java:1619)在org.apache.hadoop.hive.cli.clidriver.processlocalcmd(clidriver。java:221)在org.apache.hadoop.hive.cli.clidriver.processcmd(clidriver。java:153)在org.apache.hadoop.hive.cli.clidriver.processline(clidriver。java:364)在org.apache.hadoop.hive.cli.clidriver.executedriver(clidriver。java:712)在org.apache.hadoop.hive.cli.clidriver.run(clidriver。java:631)位于org.apache.hadoop.hive.cli.clidriver.main(clidriver。java:570)在sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)位于sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl)。java:57)在sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke(方法。java:606)在org.apache.hadoop.util.runjar.run(runjar。java:221)在org.apache.hadoop.util.runjar.main(runjar。java:136)原因:parquet.format.converter.parquetmetadataconverter.fromparquetstatistics(parquetmetadataconverter)处的java.lang.nullpointerexception。java:247)在parquet.format.converter.parquetmetadataconverter.fromparquetmetadata(parquetmetadataconverter。java:368)在parquet.format.converter.parquetmetadataconverter.readparquetmetadata(parquetmetadataconverter。java:346)在parquet.hadoop.parquetfilereader.readfooter(parquetfilereader。java:296)在parquet.hadoop.parquetfilereader.readfooter(parquetfilereader。java:254)在org.apache.hadoop.hive.ql.io.parquet.read.parquetrecordreaderwrapper.getsplit(parquetrecordreaderwrapper)。java:200)访问org.apache.hadoop.hive.ql.io.parquet.read.parquetrecordreaderwrapper.(parquetrecordreaderwrapper。java:79)访问org.apache.hadoop.hive.ql.io.parquet.read.parquetrecordreaderwrapper.(parquetrecordreaderwrapper。java:66)在org.apache.hadoop.hive.ql.io.parquet.mapredparquetinputformat.getrecordreader(mapredparquetinputformat)。java:72)在org.apache.hadoop.hive.ql.exec.fetchoperator.getrecordreader(fetchoperator)上。java:498)位于org.apache.hadoop.hive.ql.exec.fetchoperator.getnextrow(fetchoperator)。java:588) ... 还有15个
更新-在混乱了一段时间后,数据配置单元中的空值似乎被搞乱了。如何避免这种情况?
暂无答案!
目前还没有任何答案,快来回答吧!