我在独立模式下在gpu服务器上运行主服务器和1个工作服务器。提交作业后，当作业在超时之前检索并丢失执行器x次时，会发生错误taskschedulerimpl。
spark提交

spark-submit                                            \
--conf spark.plugins=com.nvidia.spark.SQLPlugin         \
--conf spark.rapids.memory.gpu.pooling.enabled=false    \
--conf spark.executor.resource.gpu.amount=1             \
--conf spark.task.resource.gpu.amount=1                 \
--jars ${SPARK_CUDF_JAR},${SPARK_RAPIDS_PLUGIN_JAR}.    \
--master spark://<ip>:7077                              \
--driver-memory 16g                                      \
--executor-memory 16g                                   \
--conf spark.cores.max=1                                \
--class com.spark.examples.Class                        \
app.jar                                                 \
-dataPath=spark/data.csv                                \
-format=csv                                             \
-numWorkers=1                                           \
-treeMethod=gpu_hist                                    \
-numRound=100                                           \
-maxDepth=8

日志

Removal of executor 1 requested
21/03/30 17:50:07 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 1
21/03/30 17:50:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
21/03/30 17:50:07 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-202103302 on worker-20330778-<ip>-33921 (<ip>:3721921) with 1 core(s)
21/03/30 17:50:07 INFO StandaloneSchedulerBackend: Granted executor ID app-2024944-000/2 on hostPort <ip>:37921 with 1 core(s), 16.0 GiB RAM
21/03/30 17:50:07 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-2044-0010/2 is now RUNNING
21/03/30 17:50:09 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<ip>:41196) with ID 2,  ResourceProfileId 0
21/03/30 17:50:09 INFO BlockManagerMasterEndpoint: Registering block manager <ip>:45111 with 9.4 GiB RAM, BlockManagerId(2, <ip>, 41351, None)
21/03/30 17:50:16 ERROR TaskSchedulerImpl: Lost executor 2 on <ip>: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
21/03/30 17:50:16 INFO DAGScheduler: Executor lost: 2 (epoch 2)

规格
我用的是aws ec2 g4dn机器。

GPU: TU104GL [Tesla T4]   
15109MiB  
Driver Version: 460.32.03  
CUDA Version: 11.2

1 worker: 1 core, 16GB of memory.