为什么我的测试集群在安全模式下运行?

dced5bon  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(441)

我正在测试一些基本的hdfs操作,比如创建目录。我的测试中有以下群集配置:

import org.apache.hadoop.fs._
import org.apache.hadoop.fs.permission.FsPermission
import org.apache.hadoop.hdfs.{HdfsConfiguration, MiniDFSCluster}

// ...

private val baseDir = new File("./target/hdfs/test").getAbsoluteFile

private val conf = new HdfsConfiguration()
conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, baseDir.getAbsolutePath)
conf.setInt("dfs.safemode.threshold.pct", 0)
private val builder = new MiniDFSCluster.Builder(conf)
private val cluster = builder.build()
cluster.waitActive()
private val fs = cluster.getFileSystem

private val host = cluster.getNameNode.getHttpAddress.getHostString
private val port = cluster.getNameNodePort

我发现,当我运行测试时,我发现我总是得到以下错误:

[warn] o.a.h.s.UserGroupInformation - PriviledgedActionException as:erip (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory [...]. Name node is in safe mode.
Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE:  If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.

不久之后。。。

[info]   org.apache.hadoop.ipc.RemoteException: Cannot create directory [...]. Name node is in safe mode.
[info] Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE:  If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.

我正在运行一个内存集群,所以我不知道为什么会看到这个。我想是在 "dfs.safemode.threshold.pct" 会阻止我根据这个答案看到这个错误,但我错了。
为什么我的内存测试集群在安全模式下运行?我怎样才能阻止它这样做?

px9o7tmv

px9o7tmv1#

问题出在 cluster.waitActive() ,它等待name节点准备就绪。这应该是 cluster.waitClusterUp() ,它显式地等待集群退出安全模式。

kmpatx3s

kmpatx3s2#

使用下面的命令从运行任何命令下面的安全模式出来-

hadoop dfsadmin -safemode leave

与safemode相关的其他命令包括

hadoop dfadmin -safemode get

获取系统当前运行状态

相关问题