使用apache ozone文件系统api会导致错误

我想上传一个文件到ApacheOzone。我使用 FileSystem 接口。java代码如下
java代码：

Configuration conf = new Configuration();
        conf.addResource(new OzoneConfiguration());

        System.setProperty("HADOOP_USER_NAME", "work");
        FileSystem fs = FileSystem.get(conf);   //"fs.defaultFS" o3fs://bucket1.volume1/

        String objectName = UUID.randomUUID().toString();
        byte[] objectData = readFile(objectName);
        Path newfilePath = new Path("/" + objectName);

        FSDataOutputStream outputStream = fs.create(newfilePath, (short) 3);
        outputStream.write(objectData);
        outputStream.flush();

        IOUtils.closeStream(outputStream);
        fs.close();

我将配置文件放在resources文件夹中。 core-site.xml 配置为：

<property>
        <name>fs.o3fs.impl</name>
        <value>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</value>
    </property>
    <property>
        <name>fs.AbstractFileSystem.o3fs.impl</name>
        <value>org.apache.hadoop.fs.ozone.OzFs</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>o3fs://bucket1.volume1/</value>
    </property>

例外情况是：

Exception in thread "main" java.lang.NullPointerException: client is null
    at java.util.Objects.requireNonNull(Objects.java:228)
    at org.apache.hadoop.hdds.scm.XceiverClientRatis.getClient(XceiverClientRatis.java:201)
    at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:227)
    at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:305)
    at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:315)
    at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:599)
    at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:452)
    at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.handleFlush(BlockOutputStream.java:463)
    at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.flush(BlockOutputStream.java:429)
    at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.flush(BlockOutputStreamEntry.java:137)
    at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleStreamAction(KeyOutputStream.java:489)
    at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:455)
    at org.apache.hadoop.ozone.client.io.KeyOutputStream.flush(KeyOutputStream.java:428)
    at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.flush(OzoneFSOutputStream.java:51)
    at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
    at java.io.DataOutputStream.flush(DataOutputStream.java:123)
    at FSTest.main(FSTest.java:28)

pom.xml文件

<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-ozone-client -->
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-ozone-client</artifactId>
    <version>0.5.0-beta</version>
</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-ozone-filesystem-lib-legacy -->
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-ozone-filesystem-lib-legacy</artifactId>
    <version>0.5.0-beta</version>
</dependency>

https://docs.alluxio.io/os/user/edge/en/ufs/ozone.html
参考此文档，ozone-0.5.0-beta有一个与scmclientconfig相关的bug，因此您必须将以下配置添加到ozone-site.xml中

<configuration>
  <property>
    <name>ozone.scm.names</name>
    <value>localhost</value>
  </property>
  <property>
    <name>scm.container.client.max.size</name>
    <value>256</value>
  </property>
  <property>
    <name>scm.container.client.idle.threshold</name>
    <value>10s</value>
  </property>
  <property>
    <name>hdds.ratis.raft.client.rpc.request.timeout</name>
    <value>60s</value>
  </property>
  <property>
    <name>hdds.ratis.raft.client.async.outstanding-requests.max</name>
    <value>32</value>
  </property>
  <property>
    <name>hdds.ratis.raft.client.rpc.watch.request.timeout</name>
    <value>180s</value>
  </property>
</configuration>

希望这能对你有所帮助。

使用apache ozone文件系统api会导致错误

1条答案

相关问题

热门标签

最新问答