emr集群上的spark步骤出错-aws

目前我正在部署一个emr群集版本5.4.0，当我尝试添加spark步骤时，它显示一个错误，并始终显示失败状态：
我试过阅读1.8.0路径的web更改java版本，但它不起作用。
这是stderr的日志。
这些是我们的版本：
释放label:emr-5.4.0 hadoop 数字化信息系统tribution:amazon 2.7.3 应用程序lications:hive 2.1.1，清管器0.16.0，Spark2.1.0
错误详细信息：线程“main”org.apache.spark.sparkeexception:应用程序\u 1507870642238 \u 0001已完成，状态为失败jar位置：command-runner.jar主类：none参数：spark submit--部署模式cluster s3://mach big data infra/inset.py失败时的操作：continue
7/10/13 05:22:45 INFO RMProxy: Connecting to ResourceManager at ip-10-5-0-147.ec2.internal/10.5.0.147:8032
        17/10/13 05:22:46 INFO Client: Requesting a new application from cluster with 1 NodeManagers
        17/10/13 05:22:46 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2048 MB per container)
        17/10/13 05:22:46 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
        17/10/13 05:22:46 INFO Client: Setting up container launch context for our AM
        17/10/13 05:22:46 INFO Client: Setting up the launch environment for our AM container
        17/10/13 05:22:46 INFO Client: Preparing resources for our AM container
        17/10/13 05:22:53 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
        17/10/13 05:23:02 INFO Client: Uploading resource file:/mnt/tmp/spark-fdeecf22-10a0-4d51-9e2f-ff63f3787b82/__spark_libs__6942539995783678180.zip -> hdfs://ip-10-5-0-147.ec2.internal:8020/user/hadoop/.sparkStaging/application_1507870642238_0001/__spark_libs__6942539995783678180.zip
        17/10/13 05:23:10 INFO Client: Uploading resource file:/etc/spark/conf/hive-site.xml -> hdfs://ip-10-5-0-147.ec2.internal:8020/user/hadoop/.sparkStaging/application_1507870642238_0001/hive-site.xml
        17/10/13 05:23:17 INFO Client: Uploading resource s3://mach-big-data-infra/ingest.py -> hdfs://ip-10-5-0-147.ec2.internal:8020/user/hadoop/.sparkStaging/application_1507870642238_0001/ingest.py
        17/10/13 05:23:17 INFO S3NativeFileSystem: Opening 's3://mach-big-data-infra/ingest.py' for reading
        17/10/13 05:23:17 INFO Client: Uploading resource file:/usr/lib/spark/python/lib/pyspark.zip -> hdfs://ip-10-5-0-147.ec2.internal:8020/user/hadoop/.sparkStaging/application_1507870642238_0001/pyspark.zip
        17/10/13 05:23:17 INFO Client: Uploading resource file:/usr/lib/spark/python/lib/py4j-0.10.4-src.zip -> hdfs://ip-10-5-0-147.ec2.internal:8020/user/hadoop/.sparkStaging/application_1507870642238_0001/py4j-0.10.4-src.zip
        17/10/13 05:23:17 INFO Client: Uploading resource file:/mnt/tmp/spark-fdeecf22-10a0-4d51-9e2f-ff63f3787b82/__spark_conf__7536775293554442846.zip -> hdfs://ip-10-5-0-147.ec2.internal:8020/user/hadoop/.sparkStaging/application_1507870642238_0001/__spark_conf__.zip
        17/10/13 05:23:18 INFO SecurityManager: Changing view acls to: hadoop
        17/10/13 05:23:18 INFO SecurityManager: Changing modify acls to: hadoop
        17/10/13 05:23:18 INFO SecurityManager: Changing view acls groups to: 
        17/10/13 05:23:18 INFO SecurityManager: Changing modify acls groups to: 
        17/10/13 05:23:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
        17/10/13 05:23:18 INFO Client: Submitting application application_1507870642238_0001 to ResourceManager
        17/10/13 05:23:18 INFO YarnClientImpl: Submitted application application_1507870642238_0001
        17/10/13 05:23:20 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:20 INFO Client: 
             client token: N/A
             diagnostics: N/A
             ApplicationMaster host: N/A
             ApplicationMaster RPC port: -1
             queue: default
             start time: 1507872198394
             final status: UNDEFINED
             tracking URL: http://ip-10-5-0-147.ec2.internal:20888/proxy/application_1507870642238_0001/
             user: hadoop
        17/10/13 05:23:21 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:22 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:23 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:24 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:25 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:26 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:27 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:28 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:29 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:30 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:31 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:32 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:33 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:34 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:35 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:36 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:37 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:38 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:39 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:40 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:41 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:42 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:43 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:44 INFO Client: Application report for application_1507870642238_0001 (state: ACCEPTED)
        17/10/13 05:23:45 INFO Client: Application report for application_1507870642238_0001 (state: RUNNING)
        17/10/13 05:23:45 INFO Client: 
             client token: N/A
             diagnostics: N/A
             ApplicationMaster host: 10.5.0.9
             ApplicationMaster RPC port: 0
             queue: default
             start time: 1507872198394
             final status: UNDEFINED
             tracking URL: http://ip-10-5-0-147.ec2.internal:20888/proxy/application_1507870642238_0001/
             user: hadoop
        17/10/13 05:23:46 INFO Client: Application report for application_1507870642238_0001 (state: RUNNING)
        17/10/13 05:23:47 INFO Client: Application report for application_1507870642238_0001 (state: RUNNING)
        17/10/13 05:23:48 INFO Client: Application report for application_1507870642238_0001 (state: RUNNING)
        17/10/13 05:23:49 INFO Client: Application report for application_1507870642238_0001 (state: RUNNING)
        17/10/13 05:23:50 INFO Client: Application report for application_1507870642238_0001 (state: RUNNING)
emr集群上的spark步骤出错-aws

暂无答案！

相关问题

热门标签

最新问答
