我在独立模式下在gpu服务器上运行主服务器和1个工作服务器。提交作业后，它在超时之前检索并丢失执行器x次。
spark提交

spark-submit                                            \
--conf spark.plugins=com.nvidia.spark.SQLPlugin         \
--conf spark.rapids.memory.gpu.pooling.enabled=false    \
--conf spark.executor.resource.gpu.amount=1             \
--conf spark.task.resource.gpu.amount=1                 \
--jars ${SPARK_CUDF_JAR},${SPARK_RAPIDS_PLUGIN_JAR}.    \
--master spark://<ip>:7077                              \
--driver-memory 2g                                      \
--executor-memory 10g                                   \
--conf spark.cores.max=1                                \
--class com.spark.examples.Class                        \
app.jar                                                 \
-dataPath=spark/data.csv                                \
-format=csv                                             \
-numWorkers=1                                           \
-treeMethod=gpu_hist                                    \
-numRound=100                                           \
-maxDepth=8

日志

StandaloneSchedulerBackend: Granted executor ID app on hostPort <ip:port> with 1 core(s), 10.0 GiB RAM
21/03/30 05:29:29 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app- is now RUNNING
21/03/30 05:29:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<ip:port>) with ID 2,  ResourceProfileId 0
21/03/30 05:29:31 INFO BlockManagerMasterEndpoint: Registering block manager (<ip:port>)with 5.8 GiB RAM, BlockManagerId(2, <ip>, 45302, None)
21/03/30 05:29:37 ERROR TaskSchedulerImpl: Lost executor 2 <ip>: Unable to create executor due to /tmp/spark-d0f43315/executor-ec041ccd/spark-295e56db/-716_cache -> ./app.jar: No space left on device
21/03/30 05:29:37 INFO DAGScheduler: Executor lost: 2 (epoch 2)

规格
我用的是aws ec2 g4dn机器。

GPU: TU104GL [Tesla T4]   
15109MiB  
Driver Version: 460.32.03  
CUDA Version: 11.2

1 worker: 1 core, 10GB of memory.