我有以下pyspark脚本,假设它连接到本地kafka集群:
from pyspark import SparkConf, SparkContext
from operator import add
import sys
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
## Constants
APP_NAME = "PythonStreamingDirectKafkaWordCount"
## OTHER FUNCTIONS/CLASSES
def main():
sc = SparkContext(appName="PythonStreamingDirectKafkaWordCount")
ssc = StreamingContext(sc, 2)
brokers, topic = sys.argv[1:]
kvs = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list": brokers})
lines = kvs.map(lambda x: x[1])
counts = lines.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a+b)
counts.pprint()
ssc.start()
ssc.awaitTermination()
if __name__ == "__main__":
main()
运行此操作时,出现以下错误:
File "/home/ubuntu/spark-1.3.0-bin-hadoop2.4/hello1.py", line 16, in main
kvs = KafkaUtils.createDirectStream(ssc, [topic], {"metadata.broker.list": brokers})
AttributeError: type object 'KafkaUtils' has no attribute 'createDirectStream'
我应该怎么做才能访问kafkautils.createdirectstream?
1条答案
按热度按时间wecizke31#
您使用的是spark 1.3.0和python版本
createDirectStream
已在spark 1.4.0中引入。spark 1.3只提供scala和java实现。如果你想使用直接流,你必须更新你的Spark安装。