\uu spark\u conf\uu.zip:没有这样的文件或目录

sf6xfgos  于 2021-05-31  发布在  Hadoop
关注(0)|答案(1)|浏览(449)

当我在hdinsight集群上提交来自azuredatafactoryv2的spark活动时,我得到了一个错误-

Application application_1524230454528_0060 failed 5 times due to AM Container for appattempt_1524230454528_0060_000005 exited with exitCode: -1000
    For more detailed output, check the application tracking page: http://hn1-rbc-cl.tsgjwtochfbexi5wvtaxuipmmg.rx.internal.cloudapp.net:8088/cluster/app/application_1524230454528_0060 Then click on links to logs of each attempt.
    Diagnostics: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
    java.io.FileNotFoundException: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatusInternal(NativeAzureFileSystem.java:2732)
    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2644)
    at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
    at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Failing this attempt. Failing the application.

有人能帮我找出问题所在吗?或者有其他解决方法可以通过azure数据工厂管道运行spark作业。我被这件事困住了,无法继续下去。根据azuredatafactory文档,我尝试将文件放在sparkjar所在容器的./files文件夹中。但仍然面临同样的问题。

pgky5nke

pgky5nke1#

你的hdi集群版本和用来开发你的程序的java版本是什么,请确保版本相同。
顺便说一句,hdi cluster 3.5+需要Java8,而早期版本需要Java7。

相关问题