试图在mesos星团上运行一个spark任务。尝试获取二进制文件时出错。
尝试打开二进制文件:
高密度光纤
从机上的本地文件系统。
用于以下路径 SPARK_EXECUTOR_URI
文件系统路径- file://home/labadmin/spark-1.2.1.tgz
```
I0501 10:27:19.302435 30510 fetcher.cpp:214] Fetching URI 'file://home/labadmin/spark-1.2.1.tgz'
Failed to fetch: file://home/labadmin/spark-1.2.1.tgz
Failed to synchronize with slave (it's probably exited)
没有端口的hdfs路径- `hdfs://ipaddress/spark/spark-1.2.1.tgz` ```
0427 09:23:21.616092 4842 fetcher.cpp:214] Fetching URI 'hdfs://ipaddress/spark/spark-1.2.1.tgz'
E0427 09:23:24.710765 4842 fetcher.cpp:113] HDFS copyToLocal failed: /usr/lib/hadoop/bin/hadoop fs -copyToLocal 'hdfs://ipaddress/spark/spark-1.2.1.tgz' '/tmp/mesos/slaves/20150427-054938-2933394698-5050-1030-S0/frameworks/20150427-054938-2933394698-5050-1030-0002/executors/20150427-054938-2933394698-5050-1030-S0/runs/5c13004a-3d8c-40a4-bac4-9c07249e1923/spark-1.2.1.tgz'
copyToLocal: Call From sclq174.lss.emc.com/ipaddress to sclq174.lss.emc.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
带端口50070的hdfs路径- hdfs://ipaddress:50070/spark/spark-1.2.1.tgz
```
I0427 13:34:25.295554 16633 fetcher.cpp:214] Fetching URI 'hdfs://ipaddress:50070/spark/spark-1.2.1.tgz'
E0427 13:34:28.438596 16633 fetcher.cpp:113] HDFS copyToLocal failed: /usr/lib/hadoop/bin/hadoop fs -copyToLocal 'hdfs://ipaddress:50070/spark/spark-1.2.1.tgz' '/tmp/mesos/slaves/20150427-054938-2933394698-5050-1030-S0/frameworks/20150427-054938-2933394698-5050-1030-0008/executors/20150427-054938-2933394698-5050-1030-S0/runs/2fc7886a-cfff-4cb2-b2f6-25988ca0f8e3/spark-1.2.1.tgz'
copyToLocal: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is:
你知道为什么它不起作用吗?
1条答案
按热度按时间kb5ga3dv1#
spark支持不同的获取二进制文件的方法:
file:
-绝对路径和file:/
uri由驱动程序的http文件服务器提供服务,每个执行器从驱动程序http服务器提取文件。hdfs:
,http:
,https:
,ftp:
-这些命令按预期从uri中下拉文件和jarlocal:
-以开头的urilocal:/
应作为本地文件存在于每个工作节点上file://home/labadmin/spark-1.2.1.tgz
无法从驱动程序访问。你可能想用local:/
uri。可能没有运行hdfs服务器
sclq174.lss.emc.com:8020
hadoop无法识别uri格式,您应该用实际的ip地址替换hostname以使其工作。192.168.1.1:50070
.