为什么spark submit包引用总是不起作用?

z8dt9xmd  于 2021-06-09  发布在  Hbase
关注(0)|答案(0)|浏览(229)

因此,我遇到了许多人正在经历的同一个问题——请看下面的错误。

WARN scheduler.TaskSetManager: Lost task 0.0 in stage 3.0 (): java.io.IOException: java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace

修复此问题的说明似乎缺失、不完整或过于复杂。我只需将它们添加到spark submit中的包列表中,就解决了其他引用问题。我也尝试过用这个htrace依赖项做同样的事情,但是没有运气。
这是我当前的spark submit命令:

spark-submit --class com.biz.HbasePassthrough \
        --packages \
            org.apache.spark:spark-streaming-kafka_2.10:1.3.0,org.apache.hbase:hbase-common:1.0.0,org.apache.hbase:hbase-client:1.0.0,org.apache.hbase:hbase-server:1.0.0,org.json4s:json4s-jackson_2.10:3.2.11,org.apache.htrace:htrace:3.1.0-incubating,org.apache.htrace:htrace-core:3.1.0-incubating,org.apache.htrace:htrace-hbase:3.1.0-incubating,org.apache.hbase:hbase-annotations:1.0.0 \
        ./my-spark_2.10-1.0.8.jar \
        >spark_log 2>&1

的输出 grep -n1 "htrace" spark_log ```
8-org.json4s#json4s-jackson_2.10 added as a dependency
9:org.apache.htrace#htrace added as a dependency
10:org.apache.htrace#htrace-core added as a dependency
11:org.apache.htrace#htrace-hbase added as a dependency
12-org.apache.hbase#hbase-annotations added as a dependency

36- found org.mortbay.jetty#jetty-util;6.1.26 in central
37: found org.apache.htrace#htrace-core;3.1.0-incubating in central
38- found org.apache.hbase#hbase-client;1.0.0 in central

75- found com.fasterxml.jackson.core#jackson-core;2.3.1 in central
76: found org.apache.htrace#htrace;3.1.0-incubating in central
77: found org.apache.htrace#htrace-hbase;3.1.0-incubating in central
78-:: resolution report :: resolve 6423ms :: artifacts dl 45ms

111- org.apache.hbase#hbase-server;1.0.0 from central in [default]
112: org.apache.htrace#htrace;3.1.0-incubating from central in [default]
113: org.apache.htrace#htrace-core;3.1.0-incubating from central in [default]
114: org.apache.htrace#htrace-hbase;3.1.0-incubating from central in [default]
115- org.apache.kafka#kafka_2.10;0.8.1.1 from central in [default]

182-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/json4s-jackson_2.10.jar at http://192.168.240.209:37644/jars/json4s-jackson_2.10.jar with timestamp 1448926760838
183:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-core.jar at http://192.168.240.209:37644/jars/htrace-core.jar with timestamp 1448926760845
184:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-hbase.jar at http://192.168.240.209:37644/jars/htrace-hbase.jar with timestamp 1448926760846
185-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/hbase-annotations.jar at http://192.168.240.209:37644/jars/hbase-annotations.jar with timestamp 1448926760847

作业开始时的所有输出似乎都表明它找到了包并将成功地使用它,但它不起作用。
我终于找到了一些我认为能让我克服这个问题的东西-这里。我通过在上面的spark submit命令中添加以下两行来修改这个命令

--driver-class-path "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar"
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \

为什么这些文件包不起作用?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题