我已经安排了一个有活力的执行者的日常工作。此作业有时运行良好,有时会随机失败,而配置没有更改。我在查看日志,但找不到任何具体内容。我正在使用以下配置:
/usr/hdp/2.6.0.3-8/spark2/bin/spark-submit --master yarn --deploy-mode cluster --driver-memory 30G --driver-cores 5 --executor-cores 4 --num-executors 30 --executor-memory 10G --conf spark.sql.files.ignoreCorruptFiles=true --conf spark.driver.maxResultSize=0 --conf spark.yarn.executor.memoryOverhead=4096 --conf spark.shuffle.service.enabled=True --conf spark.dynamicAllocation.enabled=True --conf spark.dynamicAllocation.minExecutors=30 --conf spark.dynamicAllocation.maxExecutors=80
我可以在日志中找到:
20/10/01 09:06:40 INFO Executor: Executor is trying to kill task 1455.0 in stage 8.0 (TID 43397)
20/10/01 09:06:40 INFO Executor: Executor is trying to kill task 1391.0 in stage 8.0 (TID 43293)
20/10/01 09:06:40 INFO Executor: Executor is trying to kill task 1419.0 in stage 8.0 (TID 43307)
20/10/01 09:06:40 INFO Executor: Executor is trying to kill task 1440.0 in stage 8.0 (TID 43355)
20/10/01 09:06:40 INFO Executor: Executor killed task 1440.0 in stage 8.0 (TID 43355)
20/10/01 09:06:40 INFO Executor: Executor killed task 1419.0 in stage 8.0 (TID 43307)
20/10/01 09:06:40 INFO Executor: Executor killed task 1391.0 in stage 8.0 (TID 43293)
20/10/01 09:06:40 INFO Executor: Executor killed task 1455.0 in stage 8.0 (TID 43397)
20/10/01 09:06:41 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
20/10/01 09:06:41 INFO MemoryStore: MemoryStore cleared
20/10/01 09:06:41 INFO BlockManager: BlockManager stopped
20/10/01 09:06:41 INFO ShutdownHookManager: Shutdown hook called
End of LogType:stderr
有人能帮忙找出真正的原因吗?
暂无答案!
目前还没有任何答案,快来回答吧!