spark的例子在客户端模式下抛出filenotfoundexception

wecizke3 于 2021-05-27 发布在 Hadoop

关注(0)|答案(2)|浏览(458)

我有：ubuntu14.04，hadoop2.7.7，spark2.2.0。
我刚刚安装了所有的东西。
当我尝试运行spark的示例时：

bin/spark-submit --deploy-mode client \
               --class org.apache.spark.examples.SparkPi \
               examples/jars/spark-examples_2.11-2.2.0.jar 10

我得到以下错误：
info.client:客户端令牌：n/a诊断：应用程序应用程序\u 1552490646290 \u 0007失败2次，原因是appattempt的am容器\u 1552490646290 \u 0007 \u000002退出，退出代码：-1000有关详细输出，请检查应用程序跟踪page:http：//ip-123-45-67-89:8088/cluster/app/application\u 1552490646290\u 0007然后，单击指向每次尝试日志的链接。诊断：文件：/tmp/spark-f5879f52-6777-481a-8ecf-bbb55e376901/\uu spark\u libs\uu 6948713644593068670.zip不存在java.io.filenotfoundexception:文件：/tmp/spark-f5879f52-6777-481a-8ecf-bbb55e376901/\uu spark\u libs\uu 6948713644593068670.zip不存在

at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
        at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:421)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
        at java.lang.Thread.run(Thread.java:748)

在客户机模式和集群模式下都会出现相同的错误。

hadoop ubuntu yarn apache-spark bigdata

来源：https://stackoverflow.com/questions/55157955/sparks-example-throws-filenotfoundexception-in-client-mode

2条答案

按热度按时间

bihw5rsg1#

首先，指向绑定jar的路径，包括应用程序和所有依赖项。url必须在集群内全局可见，例如，所有节点上都存在的hdfs://路径或file://路径。
第二，如果您在yarn模式下运行，那么您必须将主程序指向yarn提交应用程序，并将jar文件放在hdfs中


# Run on a YARN cluster

# Connect to a YARN cluster in client or cluster mode depending on the value

# of --deploy-mode. The cluster location will be found based on the HADOOP_CONF_DIR

# or YARN_CONF_DIR variable.

export HADOOP_CONF_DIR=XXX
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \  # can be client for client mode
hdfs://path/to/spark-examples.jar
1000

赞(0）回复(0）举报 2021-05-27

kq0g1dla2#

似乎加载spark libs失败了。丹尼尔指出，这可能与你的阅读权有关。此外，这可能与磁盘空间不足有关。
然而，在我们的例子中，为了避免传输延迟到主机和本地机器中的读/写权限，我们将spark lib放入yarn集群的hdf中，然后将它们指向 spark.yarn.archive 财产。

jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ .
export HADOOP_USER_NAME=hadoop
hadoop fs -mkdir -p /apps/spark/
hadoop fs -put -f  ${SPARK_HOME}/spark-libs.jar /apps/spark/


# spark-defaults.conf

spark.yarn.archive hdfs:///apps/spark/spark-libs.jar

赞(0）回复(0）举报 2021-05-27

我来回答

spark的例子在客户端模式下抛出filenotfoundexception

2条答案

相关问题

热门标签

最新问答