配置单元顶点失败，vertexname=map 2，同时在emr上运行，用于大文件

pvabu6sv 于 2021-06-27 发布在 Hive

关注(0)|答案(2)|浏览(522)

我正在emr集群上运行我的配置单元查询，它是一个25节点的集群，我使用了r4.4xlage来运行这个。
当我运行我的查询，我得到下面的错误。

Job Commit failed with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(java.io.IOException: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: FEAF40B78D086BEE; S3 Extended Request ID: yteHc4bRl1MrmVhqmnzm06rdzQNN8VcRwd4zqOa+rUY8m2HC2QTt9GoGR/Qu1wuJPILx4mchHRU=), S3 Extended Request ID: yteHc4bRl1MrmVhqmnzm06rdzQNN8VcRwd4zqOa+rUY8m2HC2QTt9GoGR/Qu1wuJPILx4mchHRU=)'
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.tez.TezTask
/mnt/var/lib/hadoop/steps/s-10YQZ5Z5PRUVJ/./hive-script:617:in `<main>': Error executing cmd: /usr/share/aws/emr/scripts/hive-script "--base-path" "s3://us-east-1.elasticmapreduce/libs/hive/" "--hive-versions" "latest" "--run-hive-script" "--args" "-f" "s3://205067-pcfp-app-stepfun-s3appbucket-qa/2019-02-22_App/d77a6a82-26f4-4f06-a1ea-e83677256a55/01/DeltaOutPut/processing/Scripts/script.sql" (RuntimeError)
Command exiting with ret '1'

我已经尝试设置所有Hive王参数组合如下

emrfs-site fs.s3.consistent.retryPolicyType    exponential
emrfs-site  fs.s3.consistent.metadata.tableName EmrFSMetadataAlt
emrfs-site  fs.s3.consistent.metadata.write.capacity    300
emrfs-site  fs.s3.consistent.metadata.read.capacity 600
emrfs-site  fs.s3.consistent    true
hive-site   hive.exec.stagingdir    .hive-staging
hive-site   hive.tez.java.opts  -Xmx47364m
hive-site   hive.stats.fetch.column.stats   true
hive-site   hive.stats.fetch.partition.stats    true
hive-site   hive.vectorized.execution.enabled   false
hive-site   hive.vectorized.execution.reduce.enabled    false
hive-site   tez.am.resource.memory.mb   15000
hive-site   hive.auto.convert.join  false
hive-site   hive.compute.query.using.stats  true
hive-site   hive.cbo.enable true
hive-site   tez.task.resource.memory.mb 16000

但每次都失败了。我尝试增加emr集群中节点/更大示例的数量，但结果仍然是一样的。
我也尝试过有没有tez，但还是不适合我。
这是我的示例查询。我只是复制查询的一部分

insert into filediffPcfp.TableDelta
Select rgt.FILLER1,rgt.DUNSNUMBER,rgt.BUSINESSNAME,rgt.TRADESTYLENAME,rgt.REGISTEREDADDRESSINDICATOR

请帮我找出问题所在。
添加完整Yarn原木

2019-02-26 06:28:54,318 [INFO] [TezChild] |exec.FileSinkOperator|: Final Path: FS s3://205067-pcfp-app-stepfun-s3appbucket-qa/2019-02-26_App/d996dfaa-1a62-4062-9350-d0c2bd62e867/01/DeltaOutPut/processing/Delta/.hive-staging_hive_2019-02-26_06-15-00_804_541842212852799084-1/_tmp.-ext-10000/000000_1
2019-02-26 06:28:54,319 [INFO] [TezChild] |exec.FileSinkOperator|: Writing to temp file: FS s3://205067-pcfp-app-stepfun-s3appbucket-qa/2019-02-26_App/d996dfaa-1a62-4062-9350-d0c2bd62e867/01/DeltaOutPut/processing/Delta/.hive-staging_hive_2019-02-26_06-15-00_804_541842212852799084-1/_task_tmp.-ext-10000/_tmp.000000_1
2019-02-26 06:28:54,319 [INFO] [TezChild] |exec.FileSinkOperator|: New Final Path: FS s3://205067-pcfp-app-stepfun-s3appbucket-qa/2019-02-26_App/d996dfaa-1a62-4062-9350-d0c2bd62e867/01/DeltaOutPut/processing/Delta/.hive-staging_hive_2019-02-26_06-15-00_804_541842212852799084-1/_tmp.-ext-10000/000000_1
2019-02-26 06:28:54,681 [INFO] [TezChild] |exec.FileSinkOperator|: FS[11]: records written - 1
2019-02-26 06:28:54,877 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 1000
2019-02-26 06:28:56,632 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 10000
2019-02-26 06:29:13,301 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 100000
2019-02-26 06:31:59,207 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 1000000
2019-02-26 06:34:42,686 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Received should die response from AM
2019-02-26 06:34:42,686 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Asked to die via task heartbeat
2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |task.TezTaskRunner2|: Attempting to abort attempt_1551161362408_0001_7_01_000000_1 due to an invocation of shutdownRequested
2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |tez.TezProcessor|: Received abort
2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |tez.TezProcessor|: Forwarding abort to RecordProcessor
2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |tez.MapRecordProcessor|: Forwarding abort to mapOp: {} MAP
2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |exec.MapOperator|: Received abort in operator: MAP
2019-02-26 06:34:42,705 [INFO] [TezChild] |s3.S3FSInputStream|: Encountered exception while reading '2019-02-26_App/d996dfaa-1a62-4062-9350-d0c2bd62e867/01/IncrFile/WB.ACTIVE.OCT17_01_OF_10.gz', will retry by attempting to reopen stream.
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.AbortedException: 
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
    at com.amazon.ws.emr.hadoop.fs.s3n.InputStreamWithInfo.read(InputStreamWithInfo.java:173)
    at com.amazon.ws.emr.hadoop.fs.s3.S3FSInputStream.read(S3FSInputStream.java:136)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at java.io.DataInputStream.read(DataInputStream.java:149)
    at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:179)
    at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:163)
    at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
    at java.io.InputStream.read(InputStream.java:101)
    at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:182)
    at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:218)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:176)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:255)
    at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:48)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
    at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)

Hive yarn amazon-emr hiveql hadoop2

来源：https://stackoverflow.com/questions/54779436/hive-vertex-failed-vertexname-map-2-while-running-on-emr-for-on-big-files

2条答案

按热度按时间

mdfafbf11#

让我回答我自己的问题。
在emr上运行配置单元作业时，我们注意到的第一件非常重要的事情是，步骤错误是错误的，否则将无法为您指明正确的方向。
所以最好检查Hive日志。
现在如果我们的示例被终止，那么我们就不能登录到主示例并查看日志，在这种情况下我们必须查找节点应用程序日志。
下面是如何找到节点日志的。
获取主示例id（i-04d04d09a8f7d28fd1），并用它搜索in节点。
然后打开下面的路径

/applications/hive/user/hive/hive.log.gz

在这里您可以找到预期的错误。
我们还要查找失败节点的容器日志，失败节点的详细信息可以在主示例节点中找到。

hadooplogs/j-25RSD7FFOL5JB/node/i-03f8a646a7ae97aae/daemons/

此守护进程节点日志只有在群集正在运行时才能找到，否则终止群集后emr不会将日志推送到s3日志uri中。
当我看到它时，我明白了它失败的真正原因。对我来说，这就是失败的原因
在检查主示例的示例控制器日志时，我看到有多个核心示例进入不健康状态：

2019-02-27 07:50:03,905 INFO Poller: InstanceJointStatusMap contains 21 entries (R:21):
  i-0131b7a6abd0fb8e7  1541s R   1500s ig-28 ip-10-97-51-145.tr-fr-nonprod.aws-int.thomsonreuters.com  I:   18s Y:U    81s c: 0 am:    0 H:R  0.6%Yarn unhealthy Reason : 1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers
  i-01672279d170dafd3  1539s R   1500s ig-28 ip-10-97-54-69.tr-fr-nonprod.aws-int.thomsonreuters.com  I:   16s Y:R    79s c: 0 am:241664 H:R  0.7%
  i-0227ac0f0932bd0b3  1539s R   1500s ig-28 ip-10-97-51-197.tr-fr-nonprod.aws-int.thomsonreuters.com  I:   16s Y:R    79s c: 0 am:241664 H:R  4.1%
  i-02355f335c190be40  1544s R   1500s ig-28 ip-10-97-52-150.tr-fr-nonprod.aws-int.thomsonreuters.com  I:   22s Y:R    84s c: 0 am:241664 H:R  0.2%
  i-024ed22b6affdd5ec  1540s R   1500s ig-28 ip-10-97-55-123.tr-fr-nonprod.aws-int.thomsonreuters.com  I:   16s Y:U    79s c: 0 am:    0 H:R  0.6%Yarn unhealthy Reason : 1/1 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers

一段时间后，Yarn还将核心示例列入黑名单：

2019-02-27 07:46:39,676 INFO Poller: Determining health status for App Monitor: aws157.instancecontroller.apphealth.monitor.YarnMonitor
  2019-02-27 07:46:39,688 INFO Poller: SlaveRecord i-0ac26bd7886fec338 changed state from RUNNING to BLACKLISTED
  2019-02-27 07:47:13,695 INFO Poller: SlaveRecord i-0131b7a6abd0fb8e7 changed state from RUNNING to BLACKLISTED
  2019-02-27 07:47:13,695 INFO Poller: Update SlaveRecordDbRow for i-0131b7a6abd0fb8e7 ip-10-97-51-145.tr-fr-nonprod.aws-int.thomsonreuters.com 
  2019-02-27 07:47:13,696 INFO Poller: SlaveRecord i-024ed22b6affdd5ec changed state from RUNNING to BLACKLISTED
  2019-02-27 07:47:13,696 INFO Poller: Update SlaveRecordDbRow for i-024ed22b6affdd5ec ip-10-97-55-123.tr-fr-nonprod.aws-int.thomsonreuters.com

在检查示例节点示例控制器日志时，我可以看到/mnt由于作业缓存而变满，并且使用率超过了阈值（默认情况下为90%）。
因为这种Yarn：

2019-02-27 07:40:52,231 INFO dsm-1: /mnt   total   27633 MB free    2068 MB used  25565 MB
  2019-02-27 07:40:52,231 INFO dsm-1: /      total  100663 MB free   97932 MB used   2731 MB
  2019-02-27 07:40:52,231 INFO dsm-1: cycle 17 /mnt/var/log freeSpaceMb: 2068/27633 MB freeRatio:0.07
  2019-02-27 07:40:52,248 INFO dsm-1: /mnt/var/log stats :

->在我的数据集中，源表有.gz压缩。因为.gz压缩文件是不可拆分的，因为这1个文件分配了1个Map任务。由于map任务将解压缩/mnt中的文件，因此也可能导致该问题。
->在emr中处理大量的数据需要优化一些hive属性。下面是可以在集群中设置的一些优化属性，以使查询以更好的方式运行。
v、 v.v.v.i.公司

Increase the EBS volume size for Core instances

重要的是，我们必须增加每个核心的ebs voulume，而不仅仅是主核心的ebs voulume，因为ebs volume是/mnt装载的地方，而不是在路由上。
仅此一项就解决了我的问题，但下面的配置也帮助我优化了Hive作业

hive-site.xml
                -------------
                "hive.exec.compress.intermediate" : "true",
                "hive.intermediate.compression.codec" : "org.apache.hadoop.io.compress.SnappyCodec",
                "hive.intermediate.compression.type" : "BLOCK"

                yarn-site.xml
                -------------
                "max-disk-utilization-per-disk-percentage" : "99"

这永久地解决了我的问题。
希望有人能从我的回答中受益

赞(0）回复(0）举报 2021-06-27

esbemjvw2#

从tez模式切换到mr模式，应该可以开始工作了。同时删除所有与tez相关的属性。