我有一个单节点Kafka设置。它工作得很好。然后我添加了另一个代理,并创建了一个带有两个节点的kafka集群。我没有安装单独的zookeeper和使用同一个zookeeperKafka软件包。我做了以下修改,以使集群。
两个节点中zookeeper.properties的更改:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=1000
server.1=10.20.40.120:2888:3888
server.2=10.20.40.119:2888:3888
initLimit=10
syncLimit=5
tickTime=2000
已将server.properties复制到第一个节点中的broker1.properties和第二个节点中的broker2.properties:
broker1.properties的内容:
broker.id=1
listeners=PLAINTEXT://10.20.40.120:9092
advertised.listeners=PLAINTEXT://10.20.40.120:9092
log.dirs=/tmp/kafka-logs
num.partitions=1
log.retention.hours=168
zookeeper.connect=10.20.40.120:2181,10.20.40.119:2181
zookeeper.connection.timeout.ms=6000
replica.fetch.max.bytes=4000012
message.max.bytes=2690123replica.fetch.max.bytes=4000012
message.max.bytes=2690123
max.message.bytes=4000012
第二个节点中broker2.properties的内容:
broker.id=2
listeners=PLAINTEXT://10.20.40.119:9092
advertised.listeners=PLAINTEXT://10.20.40.119:9092
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
zookeeper.connect=10.20.40.120:2181,10.20.40.119:2181
zookeeper.connection.timeout.ms=6000
在broker1中创建了一个文件myid并将1放入其中。2在broker2。
使用下面的命令启动了zookeeper:
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
在broker1和broker2中分别使用以下命令启动kafka:
nohup bin/kafka-server-start.sh config/broker1.properties &
nohup bin/kafka-server-start.sh config/broker2.properties &
现在,当我试图描述消费者补偿时,我看到如下:
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 4 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 5 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 6 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 7 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 8 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 9 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 10 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 11 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 12 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 13 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 14 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 15 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 16 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 17 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 18 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 19 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 20 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 21 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 22 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 23 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 24 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 25 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 26 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 27 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 28 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 29 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 30 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 31 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 32 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 33 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 34 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 35 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 36 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 37 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 38 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 39 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 40 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 41 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 42 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 43 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 44 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 45 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 46 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 47 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 48 Leader: 0 Replicas: 0 Isr: 0
Topic: __consumer_offsets Partition: 49 Leader: 0 Replicas: 0 Isr: 0
对于不同的分区和副本,复制参与者不应该设置为2,领导者不应该设置为1和2,isr不应该设置为1,2吗?
当我尝试创建一个消费者时,我得到了 COORDINATOR NOT AVAILABLE
以及 error_code=15
. 由于这个错误,我发现我的消费偏移量可能有问题,因此集群也有问题。
缺失的环节在哪里,如何纠正?
1条答案
按热度按时间llew8vvj1#
你应该设置
num.partitions
到代理属性文件中的2。另一件需要注意的事情是,在kafka内部创建这个主题之前,您必须确保至少有两个正在运行的代理。