尝试将数据写入hcatalog时发生invalidprotocolbufferexception(mapreduce外部)

7xzttuei  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(500)

我的测试代码:

public static void main(String [] args) throws HCatException {
    String metastoreUrl = "thrift://<censored>:9083";
    String databaseName = "default";
    String tableName = "test";

    WriteEntity.Builder builder = new WriteEntity.Builder();
    WriteEntity entity = builder.withDatabase(databaseName).withTable(tableName).build();
    HCatWriter writer = DataTransferFactory.getHCatWriter(entity, HCatalogUtils.createConfigurationMap(metastoreUrl));

    HCatRecord record1 = new DefaultHCatRecord(2);
    record1.set(0, 1);
    record1.set(1, "first line");

    List<HCatRecord> records = new ArrayList<HCatRecord>();
    records.add(record1);

    WriterContext context = writer.prepareWrite();

    writer.write(records.iterator());

    writer.commit(context);
}

当我运行这个时,我得到一个异常:

Exception in thread "main" org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat not initialized, setOutput has to be called. Cause : org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output information. Cause : java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).; Host Details : local host is: "<censored>"; destination host is: "<censored>":8020; 
at org.apache.hive.hcatalog.data.transfer.impl.HCatOutputFormatWriter.prepareWrite(HCatOutputFormatWriter.java:73)
at org.apache.hcatalog.data.MyDataTest.main(MyDataTest.java:42)
Caused by: org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output information. Cause : java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).; Host Details : local host is: "<censored>"; destination host is: "<censored>":8020; 
at org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:205)
at org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:65)
at org.apache.hive.hcatalog.data.transfer.impl.HCatOutputFormatWriter.prepareWrite(HCatOutputFormatWriter.java:66)
... 1 more
Caused by: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).; Host Details : local host is: "<censored>"; destination host is: "<censored>":8020; 
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1724)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:82)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:195)
... 3 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.<init>(RpcHeaderProtos.java:1398)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.<init>(RpcHeaderProtos.java:1362)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1492)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1487)
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)

任何可能出错的建议,或者至少我应该尝试进一步调查的建议都将非常感谢,我不知道如何离开这个地方。

7vux5j2d

7vux5j2d1#

hadoop的版本不兼容。我也遇到了这个问题。

cetgtptt

cetgtptt2#

这可能是由不兼容的protobuf消息版本造成的。我通常在使用构建在不同于集群的hadoopjar集上的客户机时看到这一点。不同的hadoop版本可以在protobuf消息中使用不同的字段。

相关问题