java在hadoop中处理内存消耗和容器数量的问题

g52tjvyc  于 2021-05-30  发布在  Hadoop
关注(0)|答案(0)|浏览(290)

我正在尝试配置hadoop集群。默认配置工作正常,但资源消耗很低。例如,从机的cpu总使用率略高于40%。我想更有效地使用集群,请参阅以下页面:
docs.hortonworks.com/hdpdocuments/hdp2/hdp-2.0.6.0/bk\u手动安装\u book/content/rpm-chap1-11.html
我有一个小型集群(1个主节点和5个从节点),每个节点都有以下硬件:
4芯
8内存gb
1个500gb硬盘
ubuntu 14.04也是如此,hadoop版本是2.5.0。
我试着计算这些值,得到:
总可用内存=8192-2024=6168(对于系统为2gb)最小容器大小=512(从上面链接中的表中提取)每个容器的内存=771->最大(最小容器大小,(总可用内存)/容器))容器数:8->最小(2*核心,总可用内存/最小容器大小)

Configuration File      Configuration Setting                   Formula                             Value
yarn-site.xml           yarn.nodemanager.resource.memory-mb     = containers * ram-per-container    6168
yarn-site.xml           yarn.scheduler.minimum-allocation-mb    = ram-per-container                 771
yarn-site.xml           yarn.scheduler.maximum-allocation-mb    = containers * ram-per-container    6168
mapred-site.xml         mapreduce.map.memory.mb                 = ram-per-container                 771
mapred-site.xml         mapreduce.reduce.memory.mb              = 2 * ram-per-container             1542
mapred-site.xml         mapreduce.map.java.opts                 = 0.8 * ram-per-container           616.8
mapred-site.xml         mapreduce.reduce.java.opts              = 0.8 * 2 * ram-per-container       1233.6
yarn-site.xml (check)   yarn.app.mapreduce.am.resource.mb       = 2 * ram-per-container             1542
yarn-site.xml (check)   yarn.app.mapreduce.am.command-opts      = 0.8 * 2 * ram-per-container       1

233.6
这个配置和它的一些变体不起作用,我得到了这样的错误

14/12/09 17:17:34 INFO mapreduce.Job: Task Id : attempt_1418155570046_0004_m_000060_1, Status : FAILED
Container [pid=5808,containerID=container_1418155570046_0004_01_000081] is running beyond physical memory limits. Current usage: 827.2 MB of 771 MB physical memory used; 1.6 GB of 1.9 GB virtual memory used. Killing container.
Dump of the process-tree for container_1418155570046_0004_01_000081 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 5815 5808 5808 5808 (java) 580 27 1706360832 211455 /usr/lib/jvm/default-java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1024M -Djava.io.tmpdir=/usr/local/hadoop/hadoop_store/tmp/nm-local-dir/usercache/hduser/appcache/application_1418155570046_0004/container_1418155570046_0004_01_000081/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1418155570046_0004/container_1418155570046_0004_01_000081 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.4.1.103 57356 attempt_1418155570046_0004_m_000060_1 81 
    |- 5808 4341 5808 5808 (bash) 0 0 14008320 303 /bin/bash -c /usr/lib/jvm/default-java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx1024M -Djava.io.tmpdir=/usr/local/hadoop/hadoop_store/tmp/nm-local-dir/usercache/hduser/appcache/application_1418155570046_0004/container_1418155570046_0004_01_000081/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1418155570046_0004/container_1418155570046_0004_01_000081 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.4.1.103 57356 attempt_1418155570046_0004_m_000060_1 81 1>/usr/local/hadoop/logs/userlogs/application_1418155570046_0004/container_1418155570046_0004_01_000081/stdout 2>/usr/local/hadoop/logs/userlogs/application_1418155570046_0004/container_1418155570046_0004_01_000081/stderr  

Container killed on request. Exit code is 143

或者我也会遇到java堆错误。
同样的应用程序在默认配置下没有失败。
那是怎么回事?
谢谢你的预付款。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题