我正在尝试将json转换为java中的parquet格式，但遇到了一个异常。
输入json:
{"list": [ {"mainBearingX": 0.178334,
"gearBoxZ": 0.03885,
"_t": 1560305236290000,
"mainBearingZ": 0.034438,
"gearBoxX": 0.035738,
"mainBearingY": 0.029445,
"gearBoxY": 0.040929,
"generatorX": 0.776837,
"generatorY": 0.124234,
"ts_id":"t1"
},
{"mainBearingX": 0.169478,
"gearBoxZ": 0.008242,
"_t": 1560305236311000,
"mainBearingZ": 0.007531,
"gearBoxX": 0.025647,
"mainBearingY": 0.029445,
"gearBoxY": 0.026282,
"generatorX": 0.770189,
"generatorY": 0.117464,
"ts_id": "t1"
}
]
}代码：
public static void toConvert(OutPut output) {
String inputFile = "test.parquetFile";
Path dataFile = new Path(inputFile);
Schema schema = ReflectData.AllowNull.get().getSchema(OutPut.class);
try (ParquetWriter<OutPut> writer = AvroParquetWriter.<OutPut>builder(dataFile)
.withSchema(schema)
.withDataModel(ReflectData.get())
.withConf(new Configuration())
.withCompressionCodec(CompressionCodecName.SNAPPY)
.withWriteMode(Mode.OVERWRITE)
.build()) {
} catch (IOException e) {
e.printStackTrace();
}
public class OutPut {
List<Map<String, Object>> list;
}例外情况：
Exception in thread "main" org.apache.parquet.schema.InvalidSchemaException: Cannot write a schema with an empty group: required group value {}
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:27)
at org.apache.parquet.schema.GroupType.accept(GroupType.java:226)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:31)
at org.apache.parquet.schema.GroupType.accept(GroupType.java:226)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:31)
at org.apache.parquet.schema.GroupType.accept(GroupType.java:226)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:31)
at org.apache.parquet.schema.GroupType.accept(GroupType.java:226)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:31)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:37)
at org.apache.parquet.schema.MessageType.accept(MessageType.java:55)
at org.apache.parquet.schema.TypeUtil.checkValidWriteSchema(TypeUtil.java:23)
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:228)
at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:273)
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:494)
1条答案
按热度按时间6ss1mwsb1#
问题是你的
OutPut类型包含类型Object作为Map:你用的是
ReflectData通过反省来推断你的类型的avro模式。然而，它不能从这些数据中推断出任何有用的东西Object类型。如果你改变了你对
OutPut使用混凝土类型，例如：那就行了。