作业成功，但sparksubmit失败

我对一份无聊的工作有意见。
我试图通过一个sparksubmit在hadoop上运行一个作业：（我注解了所有代码，以确保它不是来自作业。当然，我也尝试了所有的代码，我有同样的问题）

public static void main( String[] args )
    {
        LOGGER.info("START EXTRACT");
        LOGGER.info("args: {}", args);

        //TODO delete after tests
        LOGGER.info("END EXTRACT");
        System.exit(0);
        /* real code in commentary */
    }

当我开始工作时，我有这样一个问题：


# 10 Mar 2017 11:38:42,128  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:42 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)

# 10 Mar 2017 11:38:42,131  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:42 INFO Client:

# 10 Mar 2017 11:38:42,131  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        client token: N/A

# 10 Mar 2017 11:38:42,131  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        diagnostics: N/A

# 10 Mar 2017 11:38:42,131  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        ApplicationMaster host: N/A

# 10 Mar 2017 11:38:42,131  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        ApplicationMaster RPC port: -1

# 10 Mar 2017 11:38:42,131  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        queue: default

# 10 Mar 2017 11:38:42,131  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        start time: 1489142321100

# 10 Mar 2017 11:38:42,132  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        final status: UNDEFINED

# 10 Mar 2017 11:38:42,132  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        tracking URL: http:***************/proxy/application_1460635834146_0167/

# 10 Mar 2017 11:38:42,132  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        user: Joe

# 10 Mar 2017 11:38:43,135  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:43 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)

# 10 Mar 2017 11:38:44,138  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:44 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)

# 10 Mar 2017 11:38:45,140  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:45 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)

# 10 Mar 2017 11:38:46,144  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:46 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)

# 10 Mar 2017 11:38:47,146  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:47 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)

# 10 Mar 2017 11:38:48,148  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:48 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)

# 10 Mar 2017 11:38:49,150  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:49 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)

# 10 Mar 2017 11:38:50,153  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:50 INFO Client: Application report for application_1460635834146_0167 (state: FAILED)

# 10 Mar 2017 11:38:50,156  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:50 INFO Client:

# 10 Mar 2017 11:38:50,156  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        client token: N/A

# 10 Mar 2017 11:38:50,156  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        diagnostics: Application application_1460635834146_0167 failed 2 times due to AM Container for appattempt_1460635834146_0167_000002 exited with  exitCode: 0

# 10 Mar 2017 11:38:50,156  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: For more detailed output, check application tracking page:http://**************/cluster/app/application_1460635834146_0167Then, click on links to logs of each attempt.

# 10 Mar 2017 11:38:50,156  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: Diagnostics: Failing this attempt. Failing the application.

# 10 Mar 2017 11:38:50,157  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        ApplicationMaster host: N/A

# 10 Mar 2017 11:38:50,157  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        ApplicationMaster RPC port: -1

# 10 Mar 2017 11:38:50,157  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        queue: default

# 10 Mar 2017 11:38:50,157  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        start time: 1489142321100

# 10 Mar 2017 11:38:50,157  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        final status: FAILED

# 10 Mar 2017 11:38:50,157  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        tracking URL: http://*******************/cluster/app/application_1460635834146_0167

# 10 Mar 2017 11:38:50,157  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:        user: Joe

# 10 Mar 2017 11:38:50,158  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: Error: application failed with exception

# 10 Mar 2017 11:38:50,159  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: org.apache.spark.SparkException: Application finished with failed status

# 10 Mar 2017 11:38:50,159  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)

# 10 Mar 2017 11:38:50,159  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)

# 10 Mar 2017 11:38:50,160  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at org.apache.spark.deploy.yarn.Client.main(Client.scala)

# 10 Mar 2017 11:38:50,160  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

# 10 Mar 2017 11:38:50,160  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

# 10 Mar 2017 11:38:50,160  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

# 10 Mar 2017 11:38:50,160  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at java.lang.reflect.Method.invoke(Method.java:497)

# 10 Mar 2017 11:38:50,160  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:577)

# 10 Mar 2017 11:38:50,161  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:174)

# 10 Mar 2017 11:38:50,161  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:197)

# 10 Mar 2017 11:38:50,161  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)

# 10 Mar 2017 11:38:50,161  [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark:       at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

# 10 Mar 2017 11:38:50,174  [DEBUG] (SparkServicesImpl.java:submitSparkJobs:124): End submit SparkJob.  Exit value: 1

# 10 Mar 2017 11:38:50,174  [DEBUG] (Main.java:main:42): END sparkTest

当我在hadoop中检查日志时，我得到了：

17/03/10 11:07:56 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/03/10 11:07:57 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1460635834146_0165_000001
17/03/10 11:07:57 INFO spark.SecurityManager: Changing view acls to: yarn,Joe
17/03/10 11:07:57 INFO spark.SecurityManager: Changing modify acls to: yarn,Joe
17/03/10 11:07:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, Joe); users with modify permissions: Set(yarn, Joe)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: START EXTRACT
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: args: telrptextract-application-context.xml
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: END EXTRACT
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1460635834146_0165
End of LogType:stderr

我不明白为什么我的作业有一个0输出代码，而sparksubmit有一个错误代码。
我尝试了完整的工作，但没有成功，然后用最少的代码在工作中（如上面的代码部分所示），如果有人有想法：）谢谢！

作业成功，但sparksubmit失败

暂无答案！

相关问题

热门标签

最新问答