hadoop—Yarn中的vcore数

ajsxfq5m  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(650)

我是个毛线新手。我在我的电脑里安装了单节点hadoop,一切都是默认的。我打开resourcemanager的网页,上面说我的集群是8gbram和8个vcores。但实际上,我的计算机有3gb的ram和2个内核(我在/proc/meminfo和/proc/cpuinfo中看到了相关信息)。我想知道为什么Yarn显示太多的内存和vcore?大家能给我解释一下吗?提前谢谢!

r7xajy2e

r7xajy2e1#

yarn通过以下两个参数从yarn-site.xml conf文件中获取可用内存和内核的信息:

<!-- Default 8GB -->
<property>
  <description>Max available memory on each data node.</description>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>8192</value>
</property>

<!-- Default 8. -->
<property>
  <description>Max available cores data node.</description>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>8</value>
</property>

所以它选择8gb和8核的原因是,如果不指定任何其他内容,这些都是默认值。对于像你这样的小型机器来说,这些设置可能太高了。在小型机器上运行时,可能需要调整一些其他与内存相关的设置。
在yarn-site.xml中:

<!-- Default 1024 -->
<property>
  <description>Minimum allocation unit.</description>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>256</value>
</property>

<!-- Biggest memory allocation a container can request. Set to available memory -->
<property>
  <description>Max allocation unit.</description>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>yarnavailablememory</value>
</property>

<property>
  <description>Minimum increment setting - set to same as min-allocation</description>
  <name>yarn.scheduler.increment-allocation-mb</name>
  <value>256</value>
</property>

在mapred-site.xml中:

<!-- small cluster memory settings -->
<!-- Default 1024. Recommend setting to 4096. Should not be higher than YARN max allocation -->
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>256</value>
</property>

<!-- Default 1024. Recommend setting to 4096. Should not be higher than YARN max allocation -->
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>256</value>
</property>

<!-- Default 1536. Recommend 1024 -->
<property>
  <description>Application master allocation</description>
  <name>yarn.app.mapreduce.am.resource.mb</name>
  <value>256</value>
</property>

<!-- Recommend heapsizes to be 75% of mapreduce.map/reduce.memory.mb -->
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx204m</value>
</property>

<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx204m</value>
</property> 

<property>
  <description>Application Master JVM opts</description>
  <name>yarn.app.mapreduce.am.command-opts</name>
  <value>-Xmx204m</value>
</property>

相关问题