我正在尝试执行etl,它涉及从hdfs加载文件、应用转换并将它们写入hive。在使用sqltransforms通过遵循此文档执行转换时,我遇到了以下问题。你能帮忙吗?
java.lang.IllegalStateException: Cannot call getSchema when there is no schema
at org.apache.beam.sdk.values.PCollection.getSchema(PCollection.java:328)
at org.apache.beam.sdk.extensions.sql.impl.schema.BeamPCollectionTable.<init>(BeamPCollectionTable.java:34)
at org.apache.beam.sdk.extensions.sql.SqlTransform.toTableMap(SqlTransform.java:105)
at org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:90)
at org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:77)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:339)
at org.apache.beam.examples.SqlTest.runSqlTest(SqlTest.java:107)
at org.apache.beam.examples.SqlTest.main(SqlTest.java:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:748)
代码段:
PCollection<String> data = p.apply("ReadLines", TextIO.read().from(options.getInputFile()));
if(options.getOutput().equals("hive")){
Schema hiveTableSchema = Schema.builder()
.addStringField("eid")
.addStringField("name")
.addStringField("salary")
.addStringField("destination")
.build();
data.apply(ParDo.of(new DoFn<String, Row>() {
@ProcessElement
public void processElement(@Element String input, OutputReceiver<Row> out){
String[] values = input.split(",");
System.out.println(values);
Row row = Row.withSchema(hiveTableSchema)
.addValues(values)
.build();
out.output(row);
}
})).apply(SqlTransform.query("select eid, destination from PCOLLECTION"))
.apply(ParDo.of(new DoFn<Row, HCatRecord>() {
@ProcessElement
public void processElement(@Element Row input, OutputReceiver<HCatRecord> out){
HCatRecord record = new DefaultHCatRecord(input.getFieldCount());
for(int i=0; i < input.getFieldCount(); i++){
record.set(i, input.getString(i));
}
out.output(record);
}
}))
.apply("WriteData", HCatalogIO.write()
.withConfigProperties(configProperties)
.withDatabase("wmrpoc")
.withTable(options.getOutputTableName()));
1条答案
按热度按时间ftf50wuq1#
看起来您需要在
PCollection
. 在你链接的演练中Create...withCoder()
处理好了。在您的情况下,架构不能从您的ParDo
,beam可能看到的唯一信息是它输出类型的元素Row
但目前还没有任何信息表明ParDo
甚至对所有输出都遵循单一模式。所以你需要打电话pcollection.setRowSchema()
在你申请之前SqlTransform
告诉beam你打算从转换中得到什么样的模式ParDo
.更新
看起来你以前做的大部分事情
HCatalog
最终很可能会简化很多,例如,假设您只需要指定以下内容pipeline.apply(TextIO.readCsvRows(schema)).apply(sqlTransform)...
. 事实上,beamsql支持读取csv文件而不需要额外的转换ParDos
(通过TextTableProvider
)但它并没有连接到SqlTransform
但只能通过beamsqlcli访问