Kafka在码头集装箱里工作得很好。我可以用 docker exec -it [container name] [kafkascript]
并成功地创建主题、生成/使用消息,但当我尝试从docker容器外部使用本地kafka脚本时,我只能创建和列出主题。生成和使用消息会引发错误:
生产: ~/development/lib/kafka/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --broker-list $(docker-machine ip kafka):9092 --topic test asdf
```
[2016-09-18 10:13:48,999] ERROR Error when sending message to topic test with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for test-0
消费: `~/development/lib/kafka/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh --zookeeper $(docker-machine ip kafka):2181 --topic test --from-beginning` ```
[2016-09-18 09:57:10,389] WARN Fetching topic metadata with correlation id 0 for topics [Set(test)] from broker [BrokerEndPoint(0,ba762186182f,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-09-18 09:57:10,392] WARN [console-consumer-34526_3c15c2c24040-1474210630122-9404562b-leader-finder-thread], Failed to find leader for Set([test,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(test)] from broker [ArrayBuffer(BrokerEndPoint(0,ba762186182f,9092))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
... 3 more
我使用的是spotify/docker-kafka,但我将其升级到了0.10.0.0,并使用了jshark的一些建议来设置advised.listeners。我在mac电脑上跑步。我创造了一个叫Kafka的 Docker 机器。这是我的docker跑步:
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip kafka` --env ADVERTISED_PORT=9092 kafka
这是我的文件:
# Kafka and Zookeeper
FROM java:openjdk-8-jre
ENV DEBIAN_FRONTEND noninteractive
ENV SCALA_VERSION 2.11
ENV KAFKA_VERSION 0.10.0.0
ENV KAFKA_HOME /opt/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION"
# Install Kafka, Zookeeper and other needed things
RUN apt-get update && \
apt-get install -y zookeeper wget supervisor dnsutils && \
rm -rf /var/lib/apt/lists/* && \
apt-get clean && \
wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz && \
tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt && \
rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz
ADD scripts/start-kafka.sh /usr/bin/start-kafka.sh
# Supervisor config
ADD supervisor/kafka.conf supervisor/zookeeper.conf /etc/supervisor/conf.d/
# 2181 is zookeeper, 9092 is kafka
EXPOSE 2181
EXPOSE 9092
CMD ["supervisord", "-n"]
脚本/start-kafka.sh
3条答案
按热度按时间sqyvllje1#
这对我有用:https://stackoverflow.com/a/37655203/1839580
我的总结是
ADVERTISED_HOST
环境变量spotify/kafka
容器需要更改,这取决于您的服务是在容器内部还是外部运行。我使用docker for mac,并将docker网络设置为bridged。码头外ADVERTISED_HOST
需要设置为localhost,在docker内部,设置为myproject_kafka_1
或者你系统里的任何东西。为了解决这个问题,我在macos主机文件中添加了127.0.0.1
至myproject_kafka_1
. 我不喜欢弄乱我的主机文件,但它为我解决了这个问题。wvmv3b1j2#
在将以下条目添加到server.properties之前,我无法从容器外部与kafka进行交互:
我在这里贴了一个更完整的答案。希望别人不要像我一样花那么多时间在这件事上。
js4nwp543#
清洁剂将被设置
advertised.listeners=host-ip:port
自advertised.host.name
以及advertised.port
在Kafka中是不受欢迎的server.properties
文件。如果设置
host-ip
至0.0.0.0
它将监听来自任何地方的请求。但这是不安全的。