在使用scala运行spark作业时,获取错误“user class抛出异常:org.apache.spark.sparkexception:job aborted”

3pmvbmvn  于 2021-05-24  发布在  Spark
关注(0)|答案(0)|浏览(281)

我已经安排了一个有活力的执行者的日常工作。此作业有时运行良好,有时会随机失败,而配置没有更改。我在查看日志,但找不到任何具体内容。我正在使用以下配置:

/usr/hdp/2.6.0.3-8/spark2/bin/spark-submit --master yarn --deploy-mode cluster  --driver-memory 30G --driver-cores 5 --executor-cores 4 --num-executors 30 --executor-memory 10G  --conf spark.sql.files.ignoreCorruptFiles=true --conf spark.driver.maxResultSize=0 --conf spark.yarn.executor.memoryOverhead=4096 --conf spark.shuffle.service.enabled=True --conf spark.dynamicAllocation.enabled=True --conf spark.dynamicAllocation.minExecutors=30  --conf spark.dynamicAllocation.maxExecutors=80

我可以在日志中找到:

20/10/01 09:06:40 INFO Executor: Executor is trying to kill task 1455.0 in stage 8.0 (TID 43397)
20/10/01 09:06:40 INFO Executor: Executor is trying to kill task 1391.0 in stage 8.0 (TID 43293)
20/10/01 09:06:40 INFO Executor: Executor is trying to kill task 1419.0 in stage 8.0 (TID 43307)
20/10/01 09:06:40 INFO Executor: Executor is trying to kill task 1440.0 in stage 8.0 (TID 43355)
20/10/01 09:06:40 INFO Executor: Executor killed task 1440.0 in stage 8.0 (TID 43355)
20/10/01 09:06:40 INFO Executor: Executor killed task 1419.0 in stage 8.0 (TID 43307)
20/10/01 09:06:40 INFO Executor: Executor killed task 1391.0 in stage 8.0 (TID 43293)
20/10/01 09:06:40 INFO Executor: Executor killed task 1455.0 in stage 8.0 (TID 43397)
20/10/01 09:06:41 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
20/10/01 09:06:41 INFO MemoryStore: MemoryStore cleared
20/10/01 09:06:41 INFO BlockManager: BlockManager stopped
20/10/01 09:06:41 INFO ShutdownHookManager: Shutdown hook called

End of LogType:stderr

有人能帮忙找出真正的原因吗?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题