hadoop容器日志丢失

wljmcqd8  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(706)

我们通常可以在“/var/log/hadoop yarn/containers”路径中看到yarn容器日志。虽然我能够看到成功作业的日志,但无法看到失败作业的日志。节点管理器日志显示正在删除的日志。
日志:

2017-07-13 14:16:04,170 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor (DeletionService #1): Deleting path : /var/log/hadoop-yarn/containers/application_1234567890_12345/container_11234567890_12345_11_0000
01/stdout
2017-07-13 14:16:04,180 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl (LogAggregationService #6093): renaming /var/log/hadoop-yarn/apps/hadoop/logs/application_1234567890_12345/xx.xx.xx.xx_8041.tmp to /var/log/hadoop-yarn/apps/hadoop/logs/application_1234567890_12345/xx.xx.xx.xx_8041
2017-07-13 14:16:04,181 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor (DeletionService #3): Deleting path : /var/log/hadoop-yarn/containers/application_1234567890_12345
2017-07-13 14:16:06,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Stopping resource-monitoring for container_11234567890_12345_11_0000

下面是我的yarn-site.xml的一个片段。
有人能告诉我需要修改什么配置来保留失败作业的日志吗?

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

<property>
    <name>yarn.log.server.url</name>
    <value>http://ip-XX.XX.XX.XX:19888/jobhistory/logs</value>
</property>

<property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/mnt/yarn</value>
    <final>true</final>
</property>

<property>
    <description>Where to store container logs.</description>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/var/log/hadoop-yarn/containers</value>
</property>

<property>
    <description>Where to aggregate logs to.</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/var/log/hadoop-yarn/apps</value>
</property>

<property>
    <name>yarn.log-aggregation.enable-local-cleanup</name>
    <value>true</value>
</property>

<property>
    <name>yarn.scheduler.increment-allocation-mb</name>
    <value>32</value>
</property>

<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>
noj0wjuj

noj0wjuj1#

当日志聚合完成时,日志被移动到hdfs,通常这是hdfs上的/app日志。
检查文档中的以下设置
yarn.nodemanager.remote-app-log-dir通常/app记录在hdfs上,但在您的情况下,它被设置为/var/log/hadoop yarn/apps,这个目录在hdfs上存在吗?看起来本地目录值被错误地放在了这里。
其他可能有用的设置:
yarn.log-aggregation-enable:如果${yarn.log aggregation enable}被启用,那么nodemanager将立即将所有容器日志连接到一个文件中,并将它们上传到${yarn.nodemanager.remote app log dir}/${user.name}/logs/中的hdfs中,然后从本地userlogs目录中删除它们
yarn.nodemanager.delete.debug-delay-sec:应用程序完成后,nodemanager的deletionservice将删除应用程序的本地化文件目录和日志目录的秒数。若要诊断应用程序问题,请将此属性的值设置为足够大(例如,设置为600=10分钟),以允许检查这些目录。更改属性值后,必须重新启动nodemanager才能使其生效。

相关问题