运行hadoop需要多少内存?

o8x7eapl  于 2021-05-27  发布在  Hadoop
关注(0)|答案(0)|浏览(301)

我正在为类项目使用hadoopmapreduce,现在我正在尝试在我的windows-linux子系统上正确安装hadoop2.6.5,以便(目前)运行wordcount示例。
现在我运行两个示例文本文件,每个文件包含1个句子和5个单词,但是当我运行hadoop命令时

bin/hadoop jar wc.jar WordCount /user/joe/wordcount/input /user/joe/wordcount/output

我得到以下错误

20/09/24 12:26:28 INFO mapreduce.Job: Job job_1600964711999_0001 failed with state FAILED due to: 
Application application_1600964711999_0001 failed 2 times due to AM Container for 
appattempt_1600964711999_0001_000002 exited with  exitCode: -103
For more detailed output, check application tracking page:http://LAPTOP- 
MF1IL5JD.localdomain:8088/proxy/application_1600964711999_0001/Then, click on links to logs of each 
attempt.
Diagnostics: Container [pid=20730,containerID=container_1600964711999_0001_02_000001] is running 
beyond virtual memory limits. Current usage: 176.8 MB of 2 GB physical memory used; 566.9 GB of 4.2 
GB virtual memory used. Killing container.
Dump of the process-tree for container_1600964711999_0001_02_000001 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) 
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 20730 20728 20730 20730 (bash) 0 4 127286341632 398 /bin/bash -c /usr/lib/jvm/java-11-openjdk- 
amd64/bin/java -Dlog4j.configuration=container-log4j.properties - 
Dyarn.app.container.log.dir=/home/joe/hadoop/hadoop-

2.6.5/logs/userlogs/application_1600964711999_0001/container_1600964711999_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA  -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/home/joe/hadoop/hadoop-2.6.5/logs/userlogs/application_1600964711999_0001/container_1600964711999_0001_02_000001/stdout 2>/home/joe/hadoop/hadoop-2.6.5/logs/userlogs/application_1600964711999_0001/container_1600964711999_0001_02_000001/stderr
        |- 20738 20730 20730 20730 (java) 373 64 481410650112 44862 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/joe/hadoop/hadoop-2.6.5/logs/userlogs/application_1600964711999_0001/container_1600964711999_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
20/09/24 12:26:28 INFO mapreduce.Job: Counters: 0

现在我看到它说,当我为java提供大约4g时,它使用了超过半TB的虚拟内存,但是我应该在配置文件中更改为多少“虚拟内存”,因为我总共只有大约16g,所以1t的内存是不可能的。为什么hadoop需要这么多内存,如果我现在运行在这么小的数据集上呢?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题