shall.sh rmr brokers/topics删除主题与kafka-topics.sh上的delete标志在kafka10上的区别

axr492tv  于 2021-06-07  发布在  Kafka
关注(0)|答案(1)|浏览(377)

从我在网上和其他堆栈溢出的帖子中发现,删除Kafka主题的方法主要有两种。第一个是:a) delete.topic.enable = true 还有跑步 ./kafka-topics.sh ---delete --topic <topicName> 第二种方式: ./zookeeper-shell.sh localhost:2181 rmr brokers/topics 我注意到第一个方法标记了要删除的每个主题,过了几分钟这些主题就被删除了,而第二个方法会立即删除它们。我还注意到重新启动服务器需要几个小时,这正常吗?我在一个代理上有超过1000个主题(用于测试目的)。

z31licg0

z31licg01#

第一个方法将在zookeper中创建一个节点 admin/delete_topics/<topic> ,如果您已经像以前一样启用了主题删除,那么kafka代理(topicdeletionmanager)中的给定线程将监视 delete_topics childs将处理这个问题,这意味着从zookeper中删除日志,但也要从所有kafka副本中删除日志,确保您不会处于无效状态。整个过程描述如下:https://github.com/apache/kafka/blob/0.11.0/core/src/main/scala/kafka/controller/topicdeletionmanager.scala

/**
 * This manages the state machine for topic deletion.
 * 1. TopicCommand issues topic deletion by creating a new admin path /admin/delete_topics/<topic>
 * 2. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics
 * 3. The controller's ControllerEventThread handles topic deletion. A topic will be ineligible
 *    for deletion in the following scenarios -
  *   3.1 broker hosting one of the replicas for that topic goes down
  *   3.2 partition reassignment for partitions of that topic is in progress
 * 4. Topic deletion is resumed when -
 *    4.1 broker hosting one of the replicas for that topic is started
 *    4.2 partition reassignment for partitions of that topic completes
 * 5. Every replica for a topic being deleted is in either of the 3 states -
 *    5.1 TopicDeletionStarted Replica enters TopicDeletionStarted phase when onPartitionDeletion is invoked.
 *        This happens when the child change watch for /admin/delete_topics fires on the controller. As part of this state
 *        change, the controller sends StopReplicaRequests to all replicas. It registers a callback for the
 *        StopReplicaResponse when deletePartition=true thereby invoking a callback when a response for delete replica
 *        is received from every replica)
 *    5.2 TopicDeletionSuccessful moves replicas from
 *        TopicDeletionStarted->TopicDeletionSuccessful depending on the error codes in StopReplicaResponse
 *    5.3 TopicDeletionFailed moves replicas from
 *        TopicDeletionStarted->TopicDeletionFailed depending on the error codes in StopReplicaResponse.
 *        In general, if a broker dies and if it hosted replicas for topics being deleted, the controller marks the
 *        respective replicas in TopicDeletionFailed state in the onBrokerFailure callback. The reason is that if a
 *        broker fails before the request is sent and after the replica is in TopicDeletionStarted state,
 *        it is possible that the replica will mistakenly remain in TopicDeletionStarted state and topic deletion
 *        will not be retried when the broker comes back up.
 * 6. A topic is marked successfully deleted only if all replicas are in TopicDeletionSuccessful
 *    state. Topic deletion teardown mode deletes all topic state from the controllerContext
 *    as well as from zookeeper. This is the only time the /brokers/topics/<topic> path gets deleted. On the other hand,
 *    if no replica is in TopicDeletionStarted state and at least one replica is in TopicDeletionFailed state, then
 *    it marks the topic for deletion retry.

直接从zookeeper中删除只意味着从orchestrator中删除。当然,当请求元数据时,主题不再在这里(好吧,也许它们可以从缓存中删除),但是日志文件不会被删除(至少现在不会,我假设代理会检测到日志无效,并在某个时候删除它们),但是在代理上可能会有一些不一致(如果您正在重新平衡,你可能会弄坏很多东西)。这可能意味着一些经纪人会认为它已被删除,而另一些经纪人则认为它仍然存在。。。远非理想。
删除fom zookeeper(以及来自代理的日志)从现在看来确实是可能的,但是要注意,它可能会导致冲突、无效状态、随机错误,并且在将来的版本中可能根本不起作用。

相关问题