我在一台内存为32gb的机器上运行mapreduce作业,但是我得到了一个java堆空间错误。我已经准备好了 yarn.nodemanager.resource.memory-mb
希望我有足够的内存来运行这些任务,但我想不是。我应该如何配置mapreduce v2以避免此问题?
编辑:
16/08/30 19:00:49 INFO mapreduce.Job: Task Id : attempt_1472579604725_0003_m_000000_0, Status : FAILED
Error: Java heap space
16/08/30 19:00:55 INFO mapreduce.Job: Task Id : attempt_1472579604725_0003_m_000000_1, Status : FAILED
Error: Java heap space
16/08/30 19:01:00 INFO mapreduce.Job: Task Id : attempt_1472579604725_0003_m_000000_2, Status : FAILED
Error: Java heap space
[2] mapred-site.xml
```
[3] `yarn-site.xml` ```
<configuration>
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>
<property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property>
<property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>s8:8025</value> </property>
<property> <name>yarn.resourcemanager.scheduler.address</name> <value>s8:8030</value> </property>
<property> <name>yarn.resourcemanager.address</name> <value>s8:8032</value> </property>
<property> <name>yarn.log.server.url</name> <value>http://s8:19888/jobhistory/logs/</value> </property>
<!-- job history -->
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
<!-- proxy -->
<property><name>yarn.web-proxy.address</name><value>s8:9046</value></property>
<!-- to check the classpath in yarn, do yarn classpath -->
<!-- compress output data -->
<property><name>mapreduce.output.fileoutputformat.compress</name><value>false</value></property>
<property><name>mapred.output.fileoutputformat.compress.codec</name><value>org.apache.hadoop.io.compress.BZip2Codec</value></property>
<!-- Node configuration -->
<property> <name>yarn.nodemanager.resource.memory-mb</name> <value>33554432</value> </property>
</configuration>
1条答案
按热度按时间rkue9o1l1#
参数
yarn.nodemanager.resource.memory-mb
告诉有多少资源可用于yarn(从注解中重复)如果您想让mapreduce程序使用这些资源,您应该设置以下参数。
mapreduce.map.memory.mb内存
mapreduce.map.java.opts
mapreduce.reduce.memory.mb内存
mapreduce.reduce.java.opts文件
只需确保将java.opts设置为比memory.mb小10-20%。