更改kafka同步副本

h9a6wy2h  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(506)

我们正在运行开源的kafka confluent 5.2.1,使用avro对消息进行编码/解码。当我们构建一个新的集群并向其发布模式时,我们的 __consumer_offsets 主题具有以下配置:

$shell> kafka-topics --zookeeper localhost:2181/apps/kafka_cluster --describe --topic __consumer_offsets
Topic:__consumer_offsets        PartitionCount:50       ReplicationFactor:1     Configs:compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
        Topic: __consumer_offsets       Partition: 0    Leader: 101     Replicas: 101   Isr: 101
        Topic: __consumer_offsets       Partition: 1    Leader: 102     Replicas: 102   Isr: 102
        Topic: __consumer_offsets       Partition: 2    Leader: 101     Replicas: 101   Isr: 101
        Topic: __consumer_offsets       Partition: 3    Leader: 102     Replicas: 102   Isr: 102
...
        Topic: __consumer_offsets       Partition: 48   Leader: 101     Replicas: 101   Isr: 101
        Topic: __consumer_offsets       Partition: 49   Leader: 102     Replicas: 102   Isr: 102

然后使用以下json文件重新分配分区:

{"version":1, "partitions":[
  {"topic":"__consumer_offsets","partition":0,"replicas":[101,102,103]},
  {"topic":"__consumer_offsets","partition":1,"replicas":[102,103,101]},
  {"topic":"__consumer_offsets","partition":2,"replicas":[103,101,102]},
...
  {"topic":"__consumer_offsets","partition":45,"replicas":[101,102,103]},
  {"topic":"__consumer_offsets","partition":46,"replicas":[102,103,101]},
  {"topic":"__consumer_offsets","partition":47,"replicas":[103,101,102]},
  {"topic":"__consumer_offsets","partition":48,"replicas":[101,102,103]},
  {"topic":"__consumer_offsets","partition":49,"replicas":[102,103,101]}
]}

最终结果是副本发生更改,但同步副本有时会发生更改,有时不会:

$shell> kafka-topics --zookeeper bigdevmq02c:2181/apps/kafka_cluster --describe --topic __consumer_offsets                Topic:__consumer_offsets        PartitionCount:50       ReplicationFactor:3     Configs:compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
        Topic: __consumer_offsets       Partition: 0    Leader: 101     Replicas: 101,102,103   Isr: 101
        Topic: __consumer_offsets       Partition: 1    Leader: 102     Replicas: 102,103,101   Isr: 102,103,101
        Topic: __consumer_offsets       Partition: 2    Leader: 101     Replicas: 103,101,102   Isr: 101
        Topic: __consumer_offsets       Partition: 3    Leader: 102     Replicas: 101,102,103   Isr: 102,103,101
...
        Topic: __consumer_offsets       Partition: 48   Leader: 101     Replicas: 101,102,103   Isr: 101
        Topic: __consumer_offsets       Partition: 49   Leader: 102     Replicas: 102,103,101   Isr: 102,103,101

我希望同步副本与副本匹配,然后基于副本的第一个成员运行领导人选举,类似于: $shell> kafka-preferred-replica-election --bootstrap-server localhost:9092 但目前这种做法失败了。我做错了什么?如何纠正?
非常感谢你的帮助。
更新
我运行了一个verify,在一个空集群上运行了2.5个小时后,它仍然显示未完成:

$shell> kafka-reassign-partitions --zookeeper localhost:2181/apps/kafka_cluster --reassignment-json-file dev.json --verify
Status of partition reassignment:
Reassignment of partition __consumer_offsets-22 is still in progress
Reassignment of partition __consumer_offsets-30 is still in progress
Reassignment of partition __consumer_offsets-8 is still in progress
Reassignment of partition __consumer_offsets-21 completed successfully
4jb9z9bj

4jb9z9bj1#

环顾四周一段时间后,我们注意到集群中有一个节点运行的是kafka版本5.2.1,其余节点运行的是5.3.1。将该节点上运行的版本更改为5.3.1修复了问题。

相关问题