windows上的hadoop2.8 nodemanager正在关闭

sqserrrh  于 2021-05-27  发布在  Hadoop
关注(0)|答案(3)|浏览(647)

我已经遵循了这个网站的所有步骤。在我的电脑上一切都运行得很好,但在我的笔记本电脑上nodemanager正在关闭。
https://muhammadbilalyar.github.io/blogs/how-to-install-hadoop-on-window-10/?fbclid=iwar3u_-xwiulzm9kj7ivspd_jyqwi0jpvv9s_fhp1rppfupy3lwvrulzrr_4
以下是运行start-all.cmd命令后的nodemanager状态。

r class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.Res
ourceLocalizationService$LocalizerTracker
19/11/27 12:42:00 INFO containermanager.AuxServices: Adding auxiliary service ma
preduce_shuffle, "mapreduce_shuffle"
19/11/27 12:42:00 INFO monitor.ContainersMonitorImpl:  Using ResourceCalculatorP
lugin : org.apache.hadoop.yarn.util.ResourceCalculatorPlugin@2b72cb8a
19/11/27 12:42:00 INFO monitor.ContainersMonitorImpl:  Using ResourceCalculatorP
rocessTree : null
19/11/27 12:42:00 INFO monitor.ContainersMonitorImpl: Physical memory check enab
led: true
19/11/27 12:42:00 INFO monitor.ContainersMonitorImpl: Virtual memory check enabl
ed: true
19/11/27 12:42:00 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree curre
ntly is supported only on Linux.
19/11/27 12:42:00 INFO monitor.ContainersMonitorImpl: ContainersMonitor enabled:
 true
19/11/27 12:42:01 WARN util.SysInfoWindows: Expected split length of sysInfo to
be 11. Got 7
19/11/27 12:42:01 WARN monitor.ContainersMonitorImpl: NodeManager's totalPmem co
uld not be calculated. Setting it to -1
19/11/27 12:42:01 INFO nodemanager.NodeStatusUpdaterImpl: Nodemanager resources:
 memory set to 8192MB.
19/11/27 12:42:01 INFO nodemanager.NodeStatusUpdaterImpl: Nodemanager resources:
 vcores set to 8.
19/11/27 12:42:01 INFO nodemanager.NodeStatusUpdaterImpl: Initialized nodemanage
r with : physical-memory=8192 virtual-memory=17204 virtual-cores=8
19/11/27 12:42:01 INFO util.JvmPauseMonitor: Starting JVM pause monitor
19/11/27 12:42:01 INFO ipc.CallQueueManager: Using callQueue: class java.util.co
ncurrent.LinkedBlockingQueue queueCapacity: 2000 scheduler: class org.apache.had
oop.ipc.DefaultRpcScheduler
19/11/27 12:42:01 INFO ipc.Server: Starting Socket Reader #1 for port 51483
19/11/27 12:42:01 WARN util.SysInfoWindows: Expected split length of sysInfo to
be 11. Got 7
19/11/27 12:42:01 INFO pb.RpcServerFactoryPBImpl: Adding protocol org.apache.had
oop.yarn.api.ContainerManagementProtocolPB to the server
19/11/27 12:42:01 INFO containermanager.ContainerManagerImpl: Blocking new conta
iner-requests as container manager rpc server is still starting.
19/11/27 12:42:01 INFO ipc.Server: IPC Server Responder: starting
19/11/27 12:42:01 INFO ipc.Server: IPC Server listener on 51483: starting
19/11/27 12:42:01 INFO security.NMContainerTokenSecretManager: Updating node add
ress : annupc:51483
19/11/27 12:42:01 INFO ipc.CallQueueManager: Using callQueue: class java.util.co
ncurrent.LinkedBlockingQueue queueCapacity: 500 scheduler: class org.apache.hado
op.ipc.DefaultRpcScheduler
19/11/27 12:42:01 INFO pb.RpcServerFactoryPBImpl: Adding protocol org.apache.had
oop.yarn.server.nodemanager.api.LocalizationProtocolPB to the server
19/11/27 12:42:01 INFO localizer.ResourceLocalizationService: Localizer started
on port 8040
19/11/27 12:42:01 INFO ipc.Server: Starting Socket Reader #1 for port 8040
19/11/27 12:42:01 INFO ipc.Server: IPC Server listener on 8040: starting
19/11/27 12:42:01 INFO ipc.Server: IPC Server Responder: starting
19/11/27 12:42:01 WARN localizer.ResourceLocalizationService: Permissions incorr
ectly set for dir /tmp/hadoop-anwar/nm-local-dir/usercache, should be rwxr-xr-x,
 actual value = rwxrwxr-x
19/11/27 12:42:01 INFO localizer.ResourceLocalizationService: Attempting to init
ialize /tmp/hadoop-anwar/nm-local-dir
19/11/27 12:42:02 WARN localizer.ResourceLocalizationService: Permissions incorr
ectly set for dir /tmp/hadoop-anwar/nm-local-dir/usercache, should be rwxr-xr-x,
 actual value = rwxrwxr-x
19/11/27 12:42:02 WARN localizer.ResourceLocalizationService: Failed to setup lo
cal dir /tmp/hadoop-anwar/nm-local-dir, which was marked as good.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions incorrectly
set for dir /tmp/hadoop-anwar/nm-local-dir/usercache, should be rwxr-xr-x, actua
l value = rwxrwxr-x
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1560)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1528)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService$1.onDirsChanged(ResourceLocalizationService.java:271
)
        at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.registe
rDirsChangeListener(DirectoryCollection.java:197)
        at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.reg
isterLocalDirsChangeListener(LocalDirsHandlerService.java:242)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:371)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerM
anagerImpl.serviceStart(ContainerManagerImpl.java:490)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(No
deManager.java:369)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNod
eManager(NodeManager.java:637)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManage
r.java:684)
19/11/27 12:42:02 INFO service.AbstractService: Service org.apache.hadoop.yarn.s
erver.nodemanager.containermanager.localizer.ResourceLocalizationService failed
in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
 Failed to setup local dir /tmp/hadoop-anwar/nm-local-dir, which was marked as g
ood.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup local di
r /tmp/hadoop-anwar/nm-local-dir, which was marked as good.
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1533)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService$1.onDirsChanged(ResourceLocalizationService.java:271
)
        at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.registe
rDirsChangeListener(DirectoryCollection.java:197)
        at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.reg
isterLocalDirsChangeListener(LocalDirsHandlerService.java:242)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:371)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerM
anagerImpl.serviceStart(ContainerManagerImpl.java:490)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(No
deManager.java:369)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNod
eManager(NodeManager.java:637)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManage
r.java:684)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions i
ncorrectly set for dir /tmp/hadoop-anwar/nm-local-dir/usercache, should be rwxr-
xr-x, actual value = rwxrwxr-x
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1560)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1528)
        ... 13 more
19/11/27 12:42:02 INFO ipc.Server: Stopping server on 8040
19/11/27 12:42:02 INFO ipc.Server: Stopping IPC Server listener on 8040
19/11/27 12:42:02 INFO ipc.Server: Stopping IPC Server Responder
19/11/27 12:42:02 INFO localizer.ResourceLocalizationService: Public cache exiti
ng
19/11/27 12:42:02 INFO service.AbstractService: Service org.apache.hadoop.yarn.s
erver.nodemanager.containermanager.ContainerManagerImpl failed in state STARTED;
 cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup
local dir /tmp/hadoop-anwar/nm-local-dir, which was marked as good.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup local di
r /tmp/hadoop-anwar/nm-local-dir, which was marked as good.
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1533)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService$1.onDirsChanged(ResourceLocalizationService.java:271
)
        at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.registe
rDirsChangeListener(DirectoryCollection.java:197)
        at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.reg
isterLocalDirsChangeListener(LocalDirsHandlerService.java:242)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:371)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerM
anagerImpl.serviceStart(ContainerManagerImpl.java:490)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(No
deManager.java:369)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNod
eManager(NodeManager.java:637)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManage
r.java:684)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions i
ncorrectly set for dir /tmp/hadoop-anwar/nm-local-dir/usercache, should be rwxr-
xr-x, actual value = rwxrwxr-x
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1560)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1528)
        ... 13 more
19/11/27 12:42:02 INFO ipc.Server: Stopping server on 51483
19/11/27 12:42:02 INFO ipc.Server: Stopping IPC Server listener on 51483
19/11/27 12:42:02 INFO ipc.Server: Stopping IPC Server Responder
19/11/27 12:42:02 INFO service.AbstractService: Service NodeManager failed in st
ate STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Fail
ed to setup local dir /tmp/hadoop-anwar/nm-local-dir, which was marked as good.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup local di
r /tmp/hadoop-anwar/nm-local-dir, which was marked as good.
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1533)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService$1.onDirsChanged(ResourceLocalizationService.java:271
)
        at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.registe
rDirsChangeListener(DirectoryCollection.java:197)
        at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.reg
isterLocalDirsChangeListener(LocalDirsHandlerService.java:242)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:371)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerM
anagerImpl.serviceStart(ContainerManagerImpl.java:490)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(No
deManager.java:369)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNod
eManager(NodeManager.java:637)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManage
r.java:684)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions i
ncorrectly set for dir /tmp/hadoop-anwar/nm-local-dir/usercache, should be rwxr-
xr-x, actual value = rwxrwxr-x
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1560)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1528)
        ... 13 more
19/11/27 12:42:02 WARN nodemanager.NodeResourceMonitorImpl: org.apache.hadoop.ya
rn.server.nodemanager.NodeResourceMonitorImpl is interrupted. Exiting.
19/11/27 12:42:02 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics syst
em...
19/11/27 12:42:02 INFO impl.MetricsSystemImpl: NodeManager metrics system stoppe
d.
19/11/27 12:42:02 INFO impl.MetricsSystemImpl: NodeManager metrics system shutdo
wn complete.
19/11/27 12:42:02 FATAL nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to setup local di
r /tmp/hadoop-anwar/nm-local-dir, which was marked as good.
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1533)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService$1.onDirsChanged(ResourceLocalizationService.java:271
)
        at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.registe
rDirsChangeListener(DirectoryCollection.java:197)
        at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.reg
isterLocalDirsChangeListener(LocalDirsHandlerService.java:242)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:371)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerM
anagerImpl.serviceStart(ContainerManagerImpl.java:490)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(No
deManager.java:369)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNod
eManager(NodeManager.java:637)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManage
r.java:684)
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Permissions i
ncorrectly set for dir /tmp/hadoop-anwar/nm-local-dir/usercache, should be rwxr-
xr-x, actual value = rwxrwxr-x
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkLocalDir(ResourceLocalizationService.java:1560)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
ResourceLocalizationService.checkAndInitializeLocalDirs(ResourceLocalizationServ
ice.java:1528)
        ... 13 more
19/11/27 12:42:02 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at annupc/192.168.1.103

************************************************************/
7qhs6swi

7qhs6swi1#

你在hadoop目录中得到以下bin文件了吗
https://github.com/s911415/apache-hadoop-3.1.0-winutils
另外,请尝试清理datanode文件并从c中删除temp:\

7vhp5slm

7vhp5slm2#

它清楚地说明了关闭的原因。
org.apache.hadoop.yarn.exceptions.yarnruntimeexception:为dir/tmp/hadoop anwar/nm local dir/usercache设置的权限不正确,应为rwxr-xr-x,实际值=rwxr-x

chmod -R 755 /tmp/hadoop-anwar/nm-local-dir/usercache

我们应该解决这个问题。

zed5wv10

zed5wv103#

我有同样的问题,我认为是由于运行hadoop所需的文件夹的权限引起的。当我在administrator命令提示符中尝试相同的命令时(通过run as administrator打开),它对我很有效。如果你在ubuntu中工作,在命令之前输入sudo可能会有所帮助。

相关问题