无法在ubuntu 18.04中启动hadoop datanode hadoop 3.1.1

8dtrkrch  于 2021-05-31  发布在  Hadoop
关注(0)|答案(1)|浏览(645)

我最近在单集群模式下将hadoop3.1.1安装到我的计算机上。在完成hdfs site.xml、yarn site.xml、core site.xml、mapred-site.xml和hadoop-env.sh的所有配置之后,以下命令可以正常工作

  • 全部启动.sh
  • 停止所有.sh
  • 开始-dfs.sh
  • 开始-Yarn.sh
  • hadoop名称节点
  • hadoop namenode-格式

但我尝试执行hadoop datanode,它会给出以下错误。

2018-12-12 14:20:16,129 WARN checker.StorageLocationChecker: Exception checking StorageLocation [DISK]file:/usr/local/hadoop_store/hdfs/datanode
EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:233)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:836)
at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:508)
at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:489)
at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:511)
at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:234)
at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:141)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116)
at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:239)
at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:52)
at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-12-12 14:20:16,131 ERROR datanode.DataNode: Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:220)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2762)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2677)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2719)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2863)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2887)
2018-12-12 14:20:16,133 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
2018-12-12 14:20:16,135 INFO datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at mihiran-Inspiron-5558/127.0.1.1

为了解决这个问题,我遵循了之前那么多的问题和答案,但仍然无法解决

hdfs-site.xml    

      <configuration>
                    <property>
                              <name>dfs.replication</name>
                              <value>1</value>
                   </property>
                   <property>
                             <name>dfs.namenode.name.dir</name>
                             <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
                   </property>
                   <property>
                             <name>dfs.datanode.data.dir</name>
                             <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
                   </property>
                   <property>
                             <name>dfs.permissions</name>
                             <value>false</value>
                   </property>
            </configuration>

下面是core-site.xml的代码

core-site.xml
  <configuration>
                <property>
                    <name>fs.default.name</name>
                    <value>hdfs://localhost:9000</value>
                </property>       
  </configuration>

以下是mapred-site.xml的代码

mapred-site.xml
    <configuration>
                  <property>    
                            <name>mapred.job.tracker</name>    
                            <value>localhost:54311</value>
                  </property>    
    </configuration>

下面是yarn-site.xml的代码

yarn-site.xml 
<configuration>
    <property>
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce_shuffle</value>
   </property>
   <property>
          <name>yarn.nodemanager.aux-    services.mapreduce.shuffle.class</name>
          <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>

</configuration>
iqih9akk

iqih9akk1#

我使用此命令解决错误

sudo chown -R  mihiran  /usr/local/hadoop_store/hdfs/datanode

相关问题