在kafka代理从集群中移除并且重新分配失败之后,减少主题复制因子

mbzjlibv  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(396)

主题复制因子已增加到45,而集群中可用的kafka代理数为40。
发生这种情况的原因是重复的阻塞分区重新分配被停止。 kafka-topics --topic top --zookeeper zoo_url --describe 显示 Partition: 0 Leader: 20464 Replicas: 20464,20765,1882,20870,873,898,20752,16789,17181,20743,20854,20762,894,20459,20851,21070,20757,20766,20763,890,21173,20852,895,21314,20767,883,20467,16787,21071,20750,887,20760,7067,876,20764,891,20768,4880,20769,16788,20756,886,21172,1582,871,16827 Isr: 20464,20765,1882,20870,873,898,20752,16789,17181,20743,20762,894,20459,21070,20757,20766,20763,890,21173,895,21314,20767,883,20467,16787,20750,887,20760,7067,876,20764,891,20768,4880,20769,16788,20756,886,21172,871,16827 ... 有些副本不是群集的一部分。
跑步: kafka-reassign-partitions --zookeeper zoo_url --topics-to-move-json-file assign.json --generate --broker-list ... 失败,错误如下

Partitions reassignment failed due to replication factor: 45 larger than available brokers: 21
kafka.admin.AdminOperationException: replication factor: 45 larger than available brokers: 21
    at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:117)
    at kafka.admin.ReassignPartitionsCommand$$anonfun$generateAssignment$1.apply(ReassignPartitionsCommand.scala:110)
    at kafka.admin.ReassignPartitionsCommand$$anonfun$generateAssignment$1.apply(ReassignPartitionsCommand.scala:108)
    at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
    at kafka.admin.ReassignPartitionsCommand$.generateAssignment(ReassignPartitionsCommand.scala:108)
    at kafka.admin.ReassignPartitionsCommand$.generateAssignment(ReassignPartitionsCommand.scala:91)
    at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:50)
    at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
``` `--broker-list` 参数输入是在线代理的ID。
如何强制降低主题复制因子?
唯一有效的解决方案是通过运行 `kafka-reassign-partitions --zookeeper zoo_url --reassignment-json-file /tmp/assign.json --execute` 什么时候 `/tmp/assign.json` 就像下面一样 `{ "partitions": [ { "partition": 0, "replicas": [20743,20762,894,20459,20757,895,20467,20760], "topic": "topic" } ], "version": 1 }` 然后重新运行分区分配(分配的生成和执行)
Kafka `0.9.0.1` 作为cloudera的一部分部署。
8cdiaqws

8cdiaqws1#

可以在中添加多个分区 { "partitions": [ 列表,但使用 kafka-reassign-partitions 是处理这个问题的唯一内在方法。可能有一些外部工具可以通过编程方式生成json,但是在尝试查找当前在分配中的未知代理id时,它们也可能出错

相关问题