我面临着一个非常奇怪的问题。我有多列数据处理使用Pig。pig使用hcatalogloader在pig脚本中加载数据。这些列包含多个整数数据、字符串数据和双精度数据。其中一列（比如c1）是integer类型，无法使用parquetstorer进行存储。其他整数列没有问题，只有c1列存储失败。
错误如下：
Backend error message
---------------------
AttemptID:attempt_1413268228935_0073_m_000002_0 Info:Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Backend error message
---------------------
AttemptID:attempt_1413268228935_0073_m_000001_0 Info:Error: parquet.io.ParquetEncodingException: can not write value at 2 in tuple (,2003-11-22,840,00007,ABC,DEF,FFGG,10,0.0,0,0.0,11.11,0,7.122112,0.0,0,0.0) from type 'C1: int' to type 'optional int32 C1'
at parquet.pig.TupleWriteSupport.writeValue(TupleWriteSupport.java:199)
at parquet.pig.TupleWriteSupport.writeTuple(TupleWriteSupport.java:151)
at parquet.pig.TupleWriteSupport.write(TupleWriteSupport.java:90)
at parquet.pig.TupleWriteSupport.write(TupleWriteSupport.java:46)
at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:111)
at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:78)
at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:35)
at parquet.pig.ParquetStorer.putNext(ParquetStorer.java:121)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:635)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Number
at parquet.pig.TupleWriteSupport.writeValue(TupleWriteSupport.java:178)
... 24 more我已经描述了使用parquetstorer存储数据的别名，列c1的类型是int。parquetstorer仍然抱怨数据的类型是string，并且未能将其转换为number。
谢谢你的帮助。
1条答案
按热度按时间bwleehnv1#
我也遇到过类似的问题，我的解决方法就是将字段强制转换为chararray，然后我就可以将输出保存为parquet格式。
这是函数的源代码，http://grepcode.com/file/repo1.maven.org/maven2/com.twitter/parquet-pig/1.2.0/parquet/pig/tuplewritesupport.java
这对我来说似乎很好，但在这种情况下听起来可能是一个错误：
get（i）返回一个字符串，因此