基于Mesos的Spark簇

30byixjq 于 2021-06-26 发布在 Mesos

关注(0)|答案(1)|浏览(289)

当我尝试使用mesos的spark cluster时，作业失败了。
问题（失败的作业日志）

ser/mesos_agent/slaves/bf120f55-2e2b-4c50-8ba9-fb667f0266a7-S0/frameworks/bf120f55-2e2b-4c50-8ba9-fb667f0266a7-0001/executors/driver-20190401172856-0001/runs/fe4a8723-5663-4fed-885f-9578524bd069/spark-2.1.0-bin-hadoop2.7.tgz'
I0401 17:30:28.234169  2551 exec.cpp:161] Version: 1.0.0
I0401 17:30:28.235190  2548 exec.cpp:236] Executor registered on agent 
bf120f55-2e2b-4c50-8ba9-fb667f0266a7-S0
JAVA_HOME is not set

那是我的配置

httpserver服务器

使用python simpleserver:8080
http://simplehttpserver/spark-2.1.0-bin-hadoop2.7.tgz

spark-env.sh公司

export JAVA_HOME=/home/pduser/jdk
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI=http://192.168.44.31:8000/spark-2.1.0-bin- 
hadoop2.7.tgz

~/.bashrc

JAVA_HOME=/home/pduser/jdk
CLASSPATH=.:$JAVA_HOME/lib/tools.jar
HADOOP_HOME=/home/pduser/hadoop
export JAVA_HOME
export CLASSPATH
export HD_HOME=/home/pduser/hadoop
export HADOOP_PREFIX=$HD_HOME
export HADOOP_COMMON_HOME=$HD_HOME
export HADOOP_HDFS_HOME=$HD_HOME
export HADOOP_MAPRED_HOME=$HD_HOME
export HADOOP_YARN_HOME=$HD_HOME
export HADOOP_CONF_DIR=$HD_HOME/etc/hadoop
export YARN_CONF_DIR=$HD_HOME/etc/hadoop
export SPARK_HOME=/home/pduser/spark
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_PREFIX/bin:$SPARK_HOME/bin:
export PATH

apache-spark mesos

来源：https://stackoverflow.com/questions/55453631/spark-cluster-using-mesos