我正在尝试用mesos集群模式运行spark应用程序(我已经让客户机模式工作,但仍想尝试群集模式)
我已经启动了 spark-mesos-dispatcher
在mesos主节点上。
当我在本地路径提交程序集时 /tmp/assembly.jar
使用以下命令,
bin/spark-submit --master mesos://dispatcher:7077 --deploy-mode cluster --class com.example.Example /tmp/assembly.jar
它失败了,因为文件 /tmp/assembly.jar
在mesos从节点上不存在。
I1129 10:47:43.839771 5884 fetcher.cpp:414] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/9d725348-931a-48fb-96f7-d29a4b09f3e8-S9\/deploy","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/tmp\/assembly.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/9d725348-931a-48fb-96f7-d29a4b09f3e8-S9\/frameworks\/9d725348-931a-48fb-96f7-d29a4b09f3e8-0291\/executors\/driver-20151129104742-0008\/runs\/31bf5840-226e-4b87-ae76-d14bd2f17950","user":"user"}
I1129 10:47:43.840710 5884 fetcher.cpp:369] Fetching URI '/tmp/assembly.jar'
I1129 10:47:43.840721 5884 fetcher.cpp:243] Fetching directly into the sandbox directory
I1129 10:47:43.840731 5884 fetcher.cpp:180] Fetching URI '/tmp/assembly.jar'
I1129 10:47:43.840737 5884 fetcher.cpp:160] Copying resource with command:cp '/tmp/assembly.jar' '/var/lib/mesos/slaves/9d725348-931a-48fb-96f7-d29a4b09f3e8-S9/frameworks/9d725348-931a-48fb-96f7-d29a4b09f3e8-0291/executors/driver-20151129104742-0008/runs/31bf5840-226e-4b87-ae76-d14bd2f17950/assembly.jar'
cp: cannot stat `/tmp/assembly.jar': No such file or directory
Failed to fetch '/tmp/assembly.jar': Failed to copy with command 'cp '/tmp/assembly.jar' '/var/lib/mesos/slaves/9d725348-931a-48fb-96f7-d29a4b09f3e8-S9/frameworks/9d725348-931a-48fb-96f7-d29a4b09f3e8-0291/executors/driver-20151129104742-0008/runs/31bf5840-226e-4b87-ae76-d14bd2f17950/assembly.jar'', exit status: 256
Failed to synchronize with slave (it's probably exited)
在yarn集群模式下,spark的yarn客户机实现会将应用程序jar上传到hdfs,这样驱动程序和所有执行者都可以访问jar,但是我在restsubmissionclient中找不到这样的代码,restsubmissionclient由mesos或standalond集群模式使用。
在这种情况下谁负责上传?或者我需要手动将应用程序集放在可以通过httpuri访问的地方吗?
1条答案
按热度按时间cclgggtu1#
据我所知,你可以使用sparkcontext
addJar()
方法添加本地(到驱动程序应用程序)jar文件路径,然后将其分发到执行器节点(在客户机模式下)。当您声明要使用集群模式时,我建议您看看spark jobserver项目,它应该使spark应用程序在mesos上的运行比使用内置工具更容易。