当我尝试使用spark streaming从kafka主题中获取消息时,出现以下错误
scala> val kafkaStream = KafkaUtils.createStream(ssc, "<ipaddress>:2181","spark-streaming-consumer-group", Map("test1" -> 5))
错误:
`missing or invalid dependency detected while loading class file 'KafkaUtils.class'.
Could not access term kafka in package <root>,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of <root>.`
scala版本:2.11.8 spark版本:2.1.0.2.6.0.3-8
我用过各种各样的库来制作Kafka,但都没用:
我正在执行spark shell的代码:
./spark-shell --jars /data/home/local/504/spark-streaming-kafka_2.10-1.5.1.jar, /data/home/local/504/spark-streaming_2.10-1.5.1.jar
代码
import org.apache.spark.SparkConf
val conf = new SparkConf().setMaster("local[*]").setAppName("KafkaReceiver")
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.Seconds
val ssc = new StreamingContext(conf, Seconds(10))
import org.apache.spark.streaming.kafka.KafkaUtils
val kafkaStream = KafkaUtils.createStream(ssc, "<ipaddress>:2181","spark-streaming-consumer-group", Map("test1" -> 5))
对这个问题有什么建议吗。
1条答案
按热度按时间x6h2sr281#
既然您使用的是Scala2.11和Spark2.1.0,就应该使用这些jar
spark-streaming-kafka-0-10_2.11-2.1.0.jar
spark-streaming_2.11-2.1.0.jar
如果使用的是Kafka0.10+,则相应地进行更改。简单的程序看起来
希望这一切顺利!