此问题已在此处有答案:
Connect to Kafka running in Docker(6个回答)
三年前就关门了。
我正在尝试把Kafka和Clickhouse结合起来。我想从我的REST API发送一些数据到Kafka,并将其传递给clickhouse,这将是我的数据库。到目前为止,我能够让Kafka独立工作,并将数据从一端传递到另一端。但是当我尝试在Docker中使用它时,我遇到了一些问题。我可以看到数据在Kafka端使用,使用Kafka工具。但我的点击率显然没有得到连接。这是一个错误,当我运行我的docker时,我一直得到:
clickhouse-server_1 | %3|1584611439.583|FAIL|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
clickhouse-server_1 | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
clickhouse-server_1 | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
clickhouse-server_1 | %3|1584611439.583|FAIL|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
clickhouse-server_1 | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
clickhouse-server_1 | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
我还添加了下面的日志文件:
2020.03.19 10:16:11.637331 [ 1 ] {} <Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.637658 [ 1 ] {} <Error> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.637854 [ 1 ] {} <Error> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.678748 [ 74 ] {} <Error> void DB::DDLWorker::runMainThread(): Code: 999, e.displayText() = Coordination::Exception: All connection tries failed while connecting to ZooKeeper. Addresses: 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
(Connection loss), Stack trace (when copying this message, always include the lines below):
0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x102d352c in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f2d989 in /usr/bin/clickhouse
2. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) @ 0xdd300e4 in /usr/bin/clickhouse
3. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0xdd3070e in /usr/bin/clickhouse
4. ? @ 0xdd638b2 in /usr/bin/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::__1::vector<Poco::Net::SocketAddress, std::__1::allocator<Poco::Net::SocketAddress> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Timespan, Poco::Timespan, Poco::Timespan) @ 0xdd6099b in /usr/bin/clickhouse
6. zkutil::ZooKeeper::init(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xdd3fd53 in /usr/bin/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xdd405bd in /usr/bin/clickhouse
8. DB::Context::getZooKeeper() const @ 0xce5fb97 in /usr/bin/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0xce84c53 in /usr/bin/clickhouse
10. DB::DDLWorker::runMainThread() @ 0xce8eae3 in /usr/bin/clickhouse
11. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() const @ 0xce96031 in /usr/bin/clickhouse
12. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8f50b07 in /usr/bin/clickhouse
13. ? @ 0x8f4f00f in /usr/bin/clickhouse
14. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
15. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
(version 20.3.3.6 (official build))
以下是我正在使用的一些配置:
CREATE TABLE IF NOT EXISTS filters (
name String,
value String
) ENGINE = Kafka SETTINGS
kafka_broker_list = 'kafka://localhost:9092',
kafka_topic_list = 'testtopic',
kafka_group_name = 'test',
kafka_format = 'JSONEachRow',
kafka_num_consumers = 2
我的Kafka引导部分:
producer = KafkaProducer(bootstrap_servers=['kafka://localhost:9092'], value_serializer=lambda m: json.dumps(m).encode('ascii'))
我的Docker配置部分:
version: "3.6"
services:
clickhouse-server:
image: yandex/clickhouse-server
volumes:
- ./clickhouse/config.xml:/etc/clickhouse-server/config.xml
- ./clickhouse/zookeeper-servers.xml:/etc/clickhouse-server/conf.d/zookeeper-servers.xml
- ./shared/ch-data/clickhouse:/var/lib/clickhouse
depends_on:
- kafka
links:
- kafka
ports:
- 9000:9000
clickhouse-client:
image: yandex/clickhouse-client
entrypoint:
- /bin/sleep
command:
- infinity
kafka:
image: wurstmeister/kafka:2.11-1.0.2
volumes:
- ./shared/ch-data/kafka:/data
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: 192.168.1.86
links:
- zookeeper
ports:
- 9092:9092
- 9094:9094
zookeeper:
image: zookeeper
volumes:
- ./shared/ch-data/zookeeper:/data
ports:
- 2181:2181
我对bootstrap_Servers和Kafka_ADVERTISED_LISTENERS部分特别困惑。据我所知,我已经提供了我的主机IP作为Kafka_ADVERTISED_LISTENER,从我从here的理解
1条答案
按热度按时间yqhsw0fo1#
看来你误解了Docker网络。最重要的是,
links
功能已被弃用,所以我不确定您从哪里复制了撰写文件localhost
将始终是当前容器,而不是另一个容器修复:
kafka://localhost:9092
和bootstrap_servers=['kafka://localhost:9092']
使用实际的Docker服务名称据我所知,我提供了我的主机IP作为Kafka_ADVERTISED_LISTENER
1.只有在使用多台物理机时才需要主机IP。在同一台机器上的同一个编写文件中,clickhouse不需要这样做
1.如果您在Docker之外使用Kafka客户端,您只需要在同一物理机器上(例如,您的生产者脚本)发布localhost。
1.您应该始终为Docker网络中的客户端通告Kafka容器服务名称。在提供的文件中,这意味着您至少应该设置
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
(我不确定您使用端口9094的目的,但这是一个单独的问题)总而言之,**
KAFKA_LISTENERS
**将有3-4个端口,第四个端口用于复制的内部安全协议。你做多少广告由你决定您可以添加协议Map,如该博客文章中所讨论的。
请注意,您不需要使用任何特定的Kafka或Zookeeper Docker镜像,博客中提供的那些合成文件可以正常工作。另外,永远不要相信两个不同的Docker镜像共享同一组环境变量。
wurstmeister图像有自己的Github页面讨论侦听器