当我在hdinsight集群上提交来自azuredatafactoryv2的spark活动时,我得到了一个错误-
Application application_1524230454528_0060 failed 5 times due to AM Container for appattempt_1524230454528_0060_000005 exited with exitCode: -1000
For more detailed output, check the application tracking page: http://hn1-rbc-cl.tsgjwtochfbexi5wvtaxuipmmg.rx.internal.cloudapp.net:8088/cluster/app/application_1524230454528_0060 Then click on links to logs of each attempt.
Diagnostics: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
java.io.FileNotFoundException: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatusInternal(NativeAzureFileSystem.java:2732)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2644)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Failing this attempt. Failing the application.
有人能帮我找出问题所在吗?或者有其他解决方法可以通过azure数据工厂管道运行spark作业。我被这件事困住了,无法继续下去。根据azuredatafactory文档,我尝试将文件放在sparkjar所在容器的./files文件夹中。但仍然面临同样的问题。
1条答案
按热度按时间pgky5nke1#
你的hdi集群版本和用来开发你的程序的java版本是什么,请确保版本相同。
顺便说一句,hdi cluster 3.5+需要Java8,而早期版本需要Java7。