在hdfs中附加到现有文件时发生java异常

2w2cym1i  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(408)

我基本上是想把数据附加到hdfs中已经存在的文件中。这是我得到的例外

03:49:54,456WARN org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run:628 DataStreamer Exception
java.lang.NullPointerException
at com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
at com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$UpdatePipelineRequestProto$Builder.addAllStorageIDs(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updatePipeline(ClientNamenodeProtocolTranslatorPB.java:842)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1238)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:532)

我的复制因子是1。我使用的是apache hadoop发行版的2.5.0。这是我用来创建一个文件的代码片段,如果它不存在,或者在append模式下创建,如果它存在

String url = getHadoopUrl()+fileName;
    Path file = new Path(url);
    try {
        if(append) {
            if(hadoopFileSystem.exists(file))
                fsDataOutputStream = hadoopFileSystem.append(file);
            else
                fsDataOutputStream = hadoopFileSystem.create(file);
        }
        else
            fsDataOutputStream = hadoopFileSystem.create(file);

不太清楚是什么导致了这个异常。另外,在阅读了各种源代码之后,我很困惑hdfs是否支持附加。让我知道我错过了什么
编辑:附加我在datanode的日志中找到的堆栈跟踪

2015-10-30 16:19:54,435 INFO  org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1012136337-192.168.123.103-1411103100884:blk_1073742239_1421 src: /127.0.0.1:54160  dest: /127.0.0.1:50010
 2015-10-30 16:19:54,435 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Appending to FinalizedReplica, blk_1073742239_1421, FINALIZED
 getNumBytes()     = 812
 getBytesOnDisk()  = 812
 getVisibleLength()= 812
 getVolume()       = /Users/niranjan/hadoop/hdfs/datanode/current
 getBlockFile()    = /Users/niranjan/hadoop/hdfs/datanode/current/BP-        1012136337-192.168.123.103-1411103100884/current/finalized/blk_1073742239
 unlinked          =false
2015-10-30 16:19:54,461 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-1012136337-192.168.123.103-1411103100884:blk_1073742239_1422
java.io.IOException: Premature EOF from inputStream
    at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:435)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:693)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:569)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
    at java.lang.Thread.run(Thread.java:745)
qcbq4gxm

qcbq4gxm1#

这是hadoop从2.2.0版到2.5.1版中的一个问题,升级到更高的版本解决了这个问题,没有对配置文件进行任何调整。

dgtucam1

dgtucam12#

通过搜索我发现添加以下内容到您的 hdfs-site.xml 可能会有帮助。

<property> 
<name>dfs.datanode.max.transfer.threads</name> 
<value>8192</value> 
</property>

相关问题