跑步时 structured_kafka_wordcount.py
示例我得到以下错误:
WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
环境:hdp-2.6.5.0
我能够阅读Kafka的一个主题的内容,如 DStream
例子
运行这些作业的命令
bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.1 examples/src/main/python/sql/streaming/structured_kafka_wordcount.py ip-10-28-3-35.ec2.internal:2181 subscribe fifa2
[root@centos spark2]# bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.1 examples/src/main/python/sql/streaming/structured_kafka_wordcount.py ip-10-28-3-35.ec2.internal:2181 subscribe fifa2
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/usr/hdp/2.6.5.0-292/spark2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.spark#spark-sql-kafka-0-10_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1 in central
found org.apache.kafka#kafka-clients;0.10.0.1 in central
found net.jpountz.lz4#lz4;1.3.0 in central
found org.xerial.snappy#snappy-java;1.1.2.6 in central
found org.slf4j#slf4j-api;1.7.16 in central
found org.spark-project.spark#unused;1.0.0 in central
:: resolution report :: resolve 642ms :: artifacts dl 15ms
:: modules in use:
net.jpountz.lz4#lz4;1.3.0 from central in [default]
org.apache.kafka#kafka-clients;0.10.0.1 from central in [default]
org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1 from central in [default]
org.slf4j#slf4j-api;1.7.16 from central in [default]
org.spark-project.spark#unused;1.0.0 from central in [default]
org.xerial.snappy#snappy-java;1.1.2.6 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 6 | 0 | 0 | 0 || 6 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 6 already retrieved (0kB/17ms)
18/06/22 06:57:26 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
18/06/22 06:57:26 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
18/06/22 06:57:35 WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
18/06/22 06:57:35 WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
18/06/22 06:57:35 WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
18/06/22 06:57:35 WARN NetworkClient: Bootstrap broker ip-10-28-3-35.ec2.internal:2181 disconnected
^Z
[11]+ Stopped bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.1 examples/src/main/python/sql/streaming/structured_kafka_wordcount.py ip-10-28-3-35.ec2.internal:2181 subscribe fifa2
1条答案
按热度按时间ux6nzvsh1#
你需要提供正确的Kafka服务器,而不是Zookeeper