我们当前的项目使用jdk8、cent7、spark 2.3.1和mesos 1.4.1。我们正在升级到jdk11 headless和rhel7、spark 3.0.0和mesos 1.9.0。
在我用升级创建的新集群上，mesos会出错

I0807 16:47:34.699182 21059 executor.cpp:190] Received LAUNCH event
I0807 16:47:34.699383 21059 executor.cpp:722] Starting task driver-20200807164731-0014
I0807 16:47:34.705006 21059 executor.cpp:738] Forked command at 21068
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/var/lib/mesos-slave/slaves/38e5abe1-7fa3-431c-a8f7-c3ed02458dcf-S0/frameworks/f9ba2f70-4709-466d-8e63-baae3c7eca42-0000/executors/driver-20200807164731-0014/runs/6b1ea2c9-5a46-4c49-8f47-fafaabe9395b/spark-3.0.0-executor/jars/spark-unsafe_2.12-3.0.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Error: Failed to load class org.apache.spark.examples.SparkPi.
I0807 16:47:37.503019 21059 executor.cpp:1039] Command exited with status 101 (pid: 21068)
I0807 16:47:38.504026 21067 process.cpp:935] Stopped the socket accept loop

以下是我尝试的：
使用spark的示例代码，以防问题与我们的代码有关，但出现相同的错误。
我们通常使用spark rest api，但我也尝试了spark submit，它在deploy mode=cluster中产生了相同的错误。
如果部署模式是客户机，那么我们的产品代码和spark示例代码都将使用spark submit运行。
暂时从/tmp中删除了noexec，但仍然得到相同的错误。
暂时停止iptables。
如果作为独立jar运行，或者在跳过--deploy mode cluster选项时使用spark submit，我们的代码就会运行。
更新#1：
rest api调用（我们通常使用的）：

curl --request POST \
--url http://server1.someenv.tech:7088/v1/submissions/create \
--header 'content-type: application/json' \
--data '{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "verbose", "deploy-mode=cluster", "dataframe-active=false" ],
"appResource" : "<some artifactory url>/spark-examples_2.12-3.0.0.jar",
"clientSparkVersion" : "3.0.0",
"environmentVariables" : {
"MESOS_NATIVE_JAVA_LIBRARY" : "/usr/lib4/libmesos.so",
"MESOS_SANDBOX" : "/tmp_spark",
"SPARK_EXECUTOR_URI" : "<some artifactory url>/spark-3.0.0-executor.tar.gz"
},
"mainClass" : "org.apache.spark.examples.SparkPi",
"sparkProperties" : {
"spark.executor.extraJavaOptions" : "-Djava.io.tmpdir=/tmp_spark",
"spark.driver.extraJavaOptions" : "-Djava.io.tmpdir=/tmp_spark",
"spark.jars" : "<some artifactory url>/spark-examples_2.12-3.0.0.jar",
"spark.driver.supervise" : "false",
"spark.app.name" : "StreamingConsumer",
"spark.eventLog.enabled": "false",
"spark.submit.deployMode" : "cluster",
"spark.master" : "mesos://server1.someenv.tech:7088",
"spark.executor.memory" : "10G",
"spark.driver.memory" : "10G",
"spark.executor.cores" : 2,
"spark.local.dir":"/tmp_spark",
"spark.cores.max" : 10,
"spark.ui.enabled" : "true"
}
}'

对于spark submit（仅在调试时）

./bin/spark-submit --name <ConsumerName> --deploy-mode cluster --master mesos://<server>:<port> 
     --driver-cores 1.0 --driver-memory 10240M 
     --class com.concur.analytics.consumer.stream.Application 
     --executor-memory 10G --total-executor-cores 10 
     --conf spark.executor.extraJavaOptions=-D<envvar>=<somevalue> 
     --conf spark.master.rest.enabled=true --conf spark.eventLog.enabled=false 
     --conf spark.driver.supervise=false --conf spark.ui.enabled=true --conf spark.app.name=AppConsumer 
     --conf spark.cores.max=10 --conf spark.driver.memory=10G 
     --conf spark.driver.extraJavaOptions=-D<envvar>=<somevalue> 
     --conf spark.executor.memory=10G --conf spark.executor.cores=2 
     <myproduct jar artifactory url> <product arguments>