kafka流错误-“java.lang.noclassdeffounderror:org/apache/spark/streaming/kafka/kafkautils

rslzwgfq  于 2021-06-08  发布在  Kafka
关注(0)|答案(1)|浏览(527)

我正在eclipse中编写一个简单的kafka-spark流式代码,使用spark流式处理来自kafka代理的消息。下面是代码,我在尝试从eclipse运行代码时收到错误。
我还确保了依赖jar的位置,请帮助摆脱这个错误
对象SparkKafka流{

def main(args: Array[String]) {

val conf = new SparkConf()
  .setAppName("The swankiest Spark app ever")
  .setMaster("local[*]")

val ssc = new StreamingContext(conf, Seconds(60))
ssc.checkpoint("C:\\keerthi\\software\\eclipse-jee-mars-2-win32-  x86_64\\eclipse")

    println("Parameters:" + "zkorum:" + "group:" + "topicMap:"+"number of threads:")

val zk = "xxxxxxxx:2181"
val group = "test-consumer-group"
val topics = "my-replicated-topic"
val numThreads = 2

val topicMap =  topics.split(",").map((_,numThreads.toInt)).toMap

val lines = KafkaUtils.createStream(ssc,zk,group,topicMap).map(_._2)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x,1L)).count()

println("wordCounts:"+wordCounts)

//wordCounts.print
  }
}

例外情况:
线程“main”java.lang.noclassdeffounderror中出现异常:org/apache/spark/streaming/kafka/kafkautils$位于org.firststream.spark\u kakfa.spark\u kafka\u streaming$.main(spark\u kafka\u streaming)。scala:30)在org.firststream.spark\u kakfa.spark\u kafka\u streaming.main(spark\u kafka\u streaming.scala)上,由以下原因引起:java.lang.classnotfoundexception:org.apache.spark.streaming.kafka.kafkautils$位于java.net.urlclassloader.findclass(未知源)位于java.lang.classloader.loadclass(未知源)位于sun.misc.launcher$appclassloader.loadclass(未知源)位于java.lang.classloader.loadclass(未知源)。。。2个以上
依赖项:

<dependency>
      <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.10</artifactId>
    <version>0.8.1.1</version>
    <scope>compile</scope>
  <exclusions>
    <exclusion>
      <artifactId>jmxri</artifactId>
      <groupId>com.sun.jmx</groupId>
    </exclusion>
    <exclusion>
      <artifactId>jms</artifactId>
      <groupId>javax.jms</groupId>
    </exclusion>
    <exclusion>
      <artifactId>jmxtools</artifactId>
      <groupId>com.sun.jdmk</groupId>
    </exclusion>
  </exclusions>
 </dependency>

<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.8.2.0</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka_2.10</artifactId>
    <version>1.2.0</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>      
<version>1.2.0</version>
</dependency>
am46iovg

am46iovg1#

我评论了下面的依赖项。通过点击buildpath->configure build path->external jars,直接在eclpise中的引用库中添加了spark-streaming-kafka\u 2.10和kafka\u 2.10-0.8.1.1 jar。这就解决了问题。

<!-- dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.10</artifactId>
  <version>0.8.1.1</version>
  <scope>compile</scope>
  <exclusions>
    <exclusion>
      <artifactId>jmxri</artifactId>
      <groupId>com.sun.jmx</groupId>
    </exclusion>
    <exclusion>
      <artifactId>jms</artifactId>
      <groupId>javax.jms</groupId>
    </exclusion>
    <exclusion>
      <artifactId>jmxtools</artifactId>
      <groupId>com.sun.jdmk</groupId>
    </exclusion>
  </exclusions>
 </dependency> -->

 <!--<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.8.2.0</version>
</dependency>-->

<!-- <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka_2.10</artifactId>
    <version>1.2.0</version>
</dependency>-->

相关问题