hadoop2.6.0官方示例:在单节点设置中,yarn(mr2)比mapreduce(mr1)慢得多

daolsyd0  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(443)

我开始使用hadoop2.6.0,并根据官方文档建立了一个伪分布式单节点系统。
当我运行simplemapreduce(mr1)示例(请参阅“伪分布式操作->执行”)时,总的执行时间约为7秒。更准确地说,bash的时间给出了:

real 0m6.769s
user 0m7.375s
sys 0m0.400s

当我通过yarn(mr2)运行同一个示例时(请参阅“伪分布式操作->单节点上的yarn”),那么整个执行时间大约是100秒,因此非常慢。bash的时间给出:

real 1m38.422s
user 0m4.798s
sys 0m0.319s

因此,(出于某种原因)在用户空间之外有很大的开销。但为什么呢?
两个例子都是通过

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'

以下是pure map reduce(mr1)的更多详细信息:

(...)
15/04/10 21:12:17 INFO mapreduce.Job: Counters: 38
    File System Counters
        FILE: Number of bytes read=125642
        FILE: Number of bytes written=1009217
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=154548
        HDFS: Number of bytes written=1071
        HDFS: Number of read operations=157
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=16
    Map-Reduce Framework
        Map input records=11
        Map output records=11
        Map output bytes=263
        Map output materialized bytes=291
        Input split bytes=129
        Combine input records=0
        Combine output records=0
        Reduce input groups=5
        Reduce shuffle bytes=291
        Reduce input records=11
        Reduce output records=11
        Spilled Records=22
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=0
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
        Total committed heap usage (bytes)=1062207488
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=437
    File Output Format Counters 
        Bytes Written=197

real    0m6.769s
user    0m7.375s
sys 0m0.400s

以下是Yarn(mr2)的更多细节:

(...)
15/04/10 21:20:31 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=291
        FILE: Number of bytes written=211001
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=566
        HDFS: Number of bytes written=197
        HDFS: Number of read operations=7
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=2411
        Total time spent by all reduces in occupied slots (ms)=2717
        Total time spent by all map tasks (ms)=2411
        Total time spent by all reduce tasks (ms)=2717
        Total vcore-seconds taken by all map tasks=2411
        Total vcore-seconds taken by all reduce tasks=2717
        Total megabyte-seconds taken by all map tasks=2468864
        Total megabyte-seconds taken by all reduce tasks=2782208
    Map-Reduce Framework
        Map input records=11
        Map output records=11
        Map output bytes=263
        Map output materialized bytes=291
        Input split bytes=129
        Combine input records=0
        Combine output records=0
        Reduce input groups=5
        Reduce shuffle bytes=291
        Reduce input records=11
        Reduce output records=11
        Spilled Records=22
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=68
        CPU time spent (ms)=1160
        Physical memory (bytes) snapshot=432250880
        Virtual memory (bytes) snapshot=1719066624
        Total committed heap usage (bytes)=353370112
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=437
    File Output Format Counters 
        Bytes Written=197

real    1m38.422s
user    0m4.798s
sys 0m0.319s

有人能解释一下这个性能差距以及如何弥补它吗?

f1tvaqid

f1tvaqid1#

如果你有一个非常大的集群,并且你想将同一个集群用于不同的应用程序,比如hadoop、spark、kafka e.t.c,那么yarn就很方便了。它是为支持许多平台而设计的。我认为由于dafault配置,您能够看到时间差,我想调整集群将提供更好的性能。

相关问题