emr集群运行缓慢

x6h2sr28  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(196)

我在AmazonEMR5.5.2上运行了一个MapReduceHadoop作业,它使用Hadoop2.7.3。
我最近将emr升级到5.12.1版本,它使用hadoop2.8.0。
对于相同的输入负载,我的新集群运行速度相对较慢。
我找不出原因。也许我需要调整一些性能参数。
以下是map reduce作业计数器。看看这些计数器,有人能洞悉哪些性能参数是错误的吗?
作业计数器

File System Counters    
    FILE: Number of bytes read=1087
    FILE: Number of bytes written=24787084
    FILE: Number of read operations=0
    FILE: Number of large read operations=0
    FILE: Number of write operations=0
    HDFS: Number of bytes read=15840
    HDFS: Number of bytes written=0
    HDFS: Number of read operations=132
    HDFS: Number of large read operations=0
    HDFS: Number of write operations=0
    S3N: Number of bytes read=0
    S3N: Number of bytes written=4315
    S3N: Number of read operations=0
    S3N: Number of large read operations=0
    S3N: Number of write operations=0
Job Counters 
    Launched map tasks=132
    Launched reduce tasks=7
    Other local map tasks=132
    Total time spent by all maps in occupied slots (ms)=1576936320
    Total time spent by all reduces in occupied slots (ms)=26894720
    Total time spent by all map tasks (ms)=2463963
    Total time spent by all reduce tasks (ms)=42023
    Total vcore-milliseconds taken by all map tasks=2463963
    Total vcore-milliseconds taken by all reduce tasks=42023
    Total megabyte-milliseconds taken by all map tasks=50461962240
    Total megabyte-milliseconds taken by all reduce tasks=860631040
Map-Reduce Framework
    Map input records=12523
    Map output records=2
    Map output bytes=3236
    Map output materialized bytes=15935
    Input split bytes=15840
    Combine input records=0
    Combine output records=0
    Reduce input groups=1
    Reduce shuffle bytes=15935
    Reduce input records=2
    Reduce output records=8
    Spilled Records=4
    Shuffled Maps =924
    Failed Shuffles=0
    Merged Map outputs=924
    GC time elapsed (ms)=64327
    CPU time spent (ms)=2737480
    Physical memory (bytes) snapshot=166237839360
    Virtual memory (bytes) snapshot=2760473792512
    Total committed heap usage (bytes)=187218526208

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题