hadoop distcp由于缺少日志目录而失败

pftdvrlh  于 2021-05-31  发布在  Hadoop
关注(0)|答案(1)|浏览(466)

我正在尝试在emr群集上运行distcp命令:

hadoop distcp s3a://... hdfs://host/data/...

当我运行此命令时,会出现以下错误:

Exit code: 1
Exception message: /bin/bash: /mnt/yarn/logs/application_1524773139099_0003/container_1524773139099_0003_02_000001/stdout: No such file or directory

Stack trace: ExitCodeException exitCode=1: /bin/bash: /mnt/yarn/logs/application_1524773139099_0003/container_1524773139099_0003_02_000001/stdout: No such file or directory

at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
at org.apache.hadoop.util.Shell.run(Shell.java:479)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

我检查了集群中的所有节点,它们都有一个 /mnt/yarn/logs 我创建的目录。这是怎么回事?

hts6caw3

hts6caw31#

确保运行作业的用户有足够的权限在路径/mnt/yarn/logs中创建临时目录,如application******。最好是配置单元用户,也可以拉应用程序的Yarn日志\u 1524773139099 \u 0003,以查看可能解释实际错误的错误。

相关问题