Zookeeper scala项目中嵌入kafka的NoClassDefFoundError

agyaoht7 于 2023-04-27 发布在 Apache

关注(0)|答案(2)|浏览(178)

我试图在scala 2.11中集成一个嵌入式Kafka来进行测试，但是我得到了下面的错误，我无法理解。

java.lang.NoClassDefFoundError: org/apache/zookeeper/AsyncCallback$MultiCallback

所有版本的Kafka库都是一样的。你能告诉我我的代码中有什么问题吗？
这是我的依赖关系

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
    <version>2.4.7</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.4.0</version>
</dependency>
<dependency>
    <groupId>io.github.embeddedkafka</groupId>
    <artifactId>embedded-kafka_2.11</artifactId>
    <version>2.4.0</version>
    <scope>test</scope>
</dependency>

下面是我如何启动/停止嵌入式Kafka服务器。

implicit val embedKafkaConfig = EmbeddedKafkaConfig(kafkaPort = 7000, zooKeeperPort = 7001)
implicit val stringDeserializer = new StringDeserializer()
implicit val stringSerializer = new StringSerializer

Before("") { _: Scenario =>
  EmbeddedKafka.start()(embedKafkaConfig)
}
After("") { _: Scenario =>
  EmbeddedKafka.stop()
}

错误：

kafka.server.KafkaServer - Fatal error during KafkaServer startup. Prepare to shutdown
java.lang.NoClassDefFoundError: org/apache/zookeeper/AsyncCallback$MultiCallback
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1857)
    at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:374)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:399)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
    at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:52)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:60)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.ops.RunningKafkaOps$class.startKafka(kafkaOps.scala:88)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.EmbeddedKafka$.start(EmbeddedKafka.scala:68)

apache-zookeeper

来源：https://stackoverflow.com/questions/76109235/noclassdeffounderror-for-embedded-kafka-in-scala-project

2条答案

按热度按时间

omhiaaxx1#

The interface org.apache.zookeeper.AsyncCallback.MultiCallback appeared in Zookeeper 3.4.7+
https://github.com/apache/zookeeper/blob/release-3.4.7/src/java/main/org/apache/zookeeper/AsyncCallback.java#L271
It was absent in Zookeeper 3.4.6-
https://github.com/apache/zookeeper/blob/release-3.4.6/src/java/main/org/apache/zookeeper/AsyncCallback.java
Based on the part of pom.xml you provided, you're using Zookeeper 3.5.6 and this should be ok.
You should check whether some dependency is still using Zookeeper 3.4.6- in your actual use case.
I suspect that you provided not enough information for reproduction. I can't reproduce. https://scastie.scala-lang.org/DmytroMitin/q4BRo4taQAey47JufBc87w/1 (see also tab "Build settings"). How to create a Minimal, Reproducible Example. You can check yourself in the following way. Create an empty project with your build file and your code. If you can't reproduce the behavior there then we will not be able either.
General reasons of NoClassDefFoundError :
Why am I getting a NoClassDefFoundError in Java?
What causes and what are the differences between NoClassDefFoundError and ClassNotFoundException?
NoClassDefFoundError can signal about incompatibilities in dependencies
Unable to write DF in delta format on hdfs
ClassNotFoundException: breeze.storage.Zero$DoubleZero$
NoClassDefFoundError: scala/collection/TraversableOnce (Using Phantom Library To Fetch Data From ScyllaDB Cluster)
How to fix dependency injection when upgrading play framework from 2.5.x to 2.6.x Scala

You can do sbt dependencyTree . In Maven this should be similar to mvn dependency:tree with maven-dependency-plugin (but in sbt this doesn't show provided dependencies)

How to get a dependency tree for an artifact?

Or print System.getProperty("java.class.path") (but this shows classpath only upon JVM start).
Or add scalacOptions += "-Ylog-classpath" to build.sbt . In maven this should be similar to <arg>-Ylog-classpath</arg>

Specifying Scalac Compile-Time Option with maven-scala-plugin

Or run the following script inside your actual environment that you're using

var cl = getClass.getClassLoader
while (cl != null) {
  println(s"classloader: ${cl.getClass.getName}")
  cl match {
    case cl: URLClassLoader =>
      println("classloader urls:")
      cl.getURLs.foreach(println)
    case _ =>
      println("not URLClassLoader")
  }
  cl = cl.getParent
}

Based on the part of dependency tree you posted, for reproduction it was enough to add

<dependencies>
        ...
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.4.7</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

How can I override zookeper 3.4.6 with 3.5.6 in test scope?
Try to add to pom.xml

<dependencies>
        ...
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.5.6</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

How to force a specific version of dependency? (sbt)
Maven: how to override the dependency added by a library (maven)

赞(0）回复(0）举报 2023-04-27

kh212irz2#

I think a similar issue was earlier reported here as well, due to some conflicting versions between Kafk and zookeeper dependency versions which was reported in old Apache issues.
https://issues.apache.org/jira/browse/BIGTOP-3208
https://github.com/apache/bigtop/pull/536
I am suggesting you to generate a dependency tree and check the versions are compatible to solve the version problems.

赞(0）回复(0）举报 2023-04-27

我来回答

Zookeeper scala项目中嵌入kafka的NoClassDefFoundError

2条答案

相关问题

热门标签

最新问答