我开始使用hadoop2.6.0,并根据官方文档建立了一个伪分布式单节点系统。
当我运行simplemapreduce(mr1)示例(请参阅“伪分布式操作->执行”)时,总的执行时间约为7秒。更准确地说,bash的时间给出了:
real 0m6.769s
user 0m7.375s
sys 0m0.400s
当我通过yarn(mr2)运行同一个示例时(请参阅“伪分布式操作->单节点上的yarn”),那么整个执行时间大约是100秒,因此非常慢。bash的时间给出:
real 1m38.422s
user 0m4.798s
sys 0m0.319s
因此,(出于某种原因)在用户空间之外有很大的开销。但为什么呢?
两个例子都是通过
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'
以下是pure map reduce(mr1)的更多详细信息:
(...)
15/04/10 21:12:17 INFO mapreduce.Job: Counters: 38
File System Counters
FILE: Number of bytes read=125642
FILE: Number of bytes written=1009217
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=154548
HDFS: Number of bytes written=1071
HDFS: Number of read operations=157
HDFS: Number of large read operations=0
HDFS: Number of write operations=16
Map-Reduce Framework
Map input records=11
Map output records=11
Map output bytes=263
Map output materialized bytes=291
Input split bytes=129
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=291
Reduce input records=11
Reduce output records=11
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1062207488
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=437
File Output Format Counters
Bytes Written=197
real 0m6.769s
user 0m7.375s
sys 0m0.400s
以下是Yarn(mr2)的更多细节:
(...)
15/04/10 21:20:31 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=291
FILE: Number of bytes written=211001
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=566
HDFS: Number of bytes written=197
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2411
Total time spent by all reduces in occupied slots (ms)=2717
Total time spent by all map tasks (ms)=2411
Total time spent by all reduce tasks (ms)=2717
Total vcore-seconds taken by all map tasks=2411
Total vcore-seconds taken by all reduce tasks=2717
Total megabyte-seconds taken by all map tasks=2468864
Total megabyte-seconds taken by all reduce tasks=2782208
Map-Reduce Framework
Map input records=11
Map output records=11
Map output bytes=263
Map output materialized bytes=291
Input split bytes=129
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=291
Reduce input records=11
Reduce output records=11
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=68
CPU time spent (ms)=1160
Physical memory (bytes) snapshot=432250880
Virtual memory (bytes) snapshot=1719066624
Total committed heap usage (bytes)=353370112
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=437
File Output Format Counters
Bytes Written=197
real 1m38.422s
user 0m4.798s
sys 0m0.319s
有人能解释一下这个性能差距以及如何弥补它吗?
1条答案
按热度按时间f1tvaqid1#
如果你有一个非常大的集群,并且你想将同一个集群用于不同的应用程序,比如hadoop、spark、kafka e.t.c,那么yarn就很方便了。它是为支持许多平台而设计的。我认为由于dafault配置,您能够看到时间差,我想调整集群将提供更好的性能。