Kafka broker重新启动“复制因子:3大于经纪人0,”

qxgroojn  于 2023-06-21  发布在  Apache
关注(0)|答案(1)|浏览(437)

我们在confluent-platform 7.0.1中有8个Kafka brokers集群和5个zookeeper集群。例如,当我们重新启动brokerid 2时,根据Kafka日志为fallowing

[2023-06-15 09:56:09,906] INFO Socket connection established, initiating session, client: /10.136.132.61:36972, server: datagovstg-kfk04.deltaww.com/10.136.132.62:2181 (org.apache.zookeeper.ClientCnxn)
[2023-06-15 09:56:10,013] INFO Session establishment complete on server datagovstg-kfk04.deltaww.com/10.136.132.62:2181, session id = 0x535f9587f0000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2023-06-15 09:56:10,014] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2023-06-15 09:56:11,283] INFO [feature-zk-node-event-process-thread]: Starting (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2023-06-15 09:56:11,483] INFO Updated cache from existing <empty> to latest FinalizedFeaturesAndEpoch(features=Features{}, epoch=0). (kafka.server.FinalizedFeatureCache)
[2023-06-15 09:56:11,490] INFO Cluster ID = Xugt0nfcSjW7zXrDhEzkuA (kafka.server.KafkaServer)

// DELETED

[2023-06-15 09:56:13,644] INFO Creating /brokers/ids/2 (is it secure? false) (kafka.zk.KafkaZkClient)
[2023-06-15 09:56:13,732] INFO Stat of the created znode at /brokers/ids/2 is: 85899346228,85899346228,1686794161605,1686794161605,1,0,0,1466719931400192,329,0,85899346228
 (kafka.zk.KafkaZkClient)
[2023-06-15 09:56:13,732] INFO Registered broker 2 at path /brokers/ids/2 with addresses: PLAINTEXT://datagovstg-kfk03.deltaww.com:9092,SASL_PLAINTEXT://datagovstg-kfk03.deltaww.com:9093, czxid (broker epoch): 85899346228 (kafka.zk.KafkaZkClient)
[2023-06-15 09:56:13,789] INFO [ExpirationReaper-2-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2023-06-15 09:56:13,793] INFO [ExpirationReaper-2-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2023-06-15 09:56:13,794] INFO [ExpirationReaper-2-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2023-06-15 09:56:13,809] INFO [GroupCoordinator 2]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2023-06-15 09:56:13,818] INFO [GroupCoordinator 2]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2023-06-15 09:56:13,834] INFO [TransactionCoordinator id=2] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2023-06-15 09:56:13,838] INFO [Transaction Marker Channel Manager 2]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2023-06-15 09:56:13,838] INFO [TransactionCoordinator id=2] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)

// DELETED

[2023-06-15 09:56:21,639] INFO Kafka version: 7.0.1-ccs (org.apache.kafka.common.utils.AppInfoParser)
[2023-06-15 09:56:21,639] INFO Kafka commitId: b7e52413e7cb3e8b (org.apache.kafka.common.utils.AppInfoParser)
[2023-06-15 09:56:21,639] INFO Kafka startTimeMs: 1686794181632 (org.apache.kafka.common.utils.AppInfoParser)
[2023-06-15 09:56:21,641] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)
[2023-06-15 09:56:21,940] INFO [Admin Manager on Broker 2]: Error processing create topic request CreatableTopic(name='_schemas', numPartitions=3, replicationFactor=3, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 0.
[2023-06-15 09:56:21,994] INFO [Admin Manager on Broker 2]: Error processing create topic request CreatableTopic(name='__consumer_offsets', numPartitions=50, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='compression.type', value='producer'), CreateableTopicConfig(name='cleanup.policy', value='compact'), CreateableTopicConfig(name='segment.bytes', value='104857600')]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 0.
[2023-06-15 09:56:22,095] INFO [Admin Manager on Broker 2]: Error processing create topic request CreatableTopic(name='_schemas', numPartitions=3, replicationFactor=3, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 0.
[2023-06-15 09:56:22,136] INFO [Admin Manager on Broker 2]: Error processing create topic request CreatableTopic(name='_schemas', numPartitions=3, replicationFactor=3, assignments=[], configs=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 0.
[2023-06-15 09:56:22,174] INFO [Admin Manager on Broker 2]: Error processing create topic request CreatableTopic(name='__consumer_offsets', numPartitions=50, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='compression.type', value='producer'), CreateableTopicConfig(name='cleanup.policy', value='compact'), CreateableTopicConfig(name='segment.bytes', value='104857600')]) (kafka.server.ZkAdminManager)

看起来brokerid 2有加入集群,但是

org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 0.

我用bootstrap.server= broker 1写了一个简单的java程序来描述集群,我可以在可用的节点中得到broker 2。但是我不能使用bootstrap.server= broker 2来描述集群或列表主题。我使用zookeeper-shell连接到zookeeper,我使用这个命令ls /brokers/ids,然后我在孩子中获得八个代理ID。当我关闭所有Kafka broker并重新启动kafka broker时,这种情况就消失了。我不认为这是处理此错误的正确解决方案。
我们有一些主题只有一个副本。一位同事认为这是根本原因,并建议我们应该将每个主题的复制因子设置为3。应重新分配现有主题,使其复制系数为3。这是我们遇到的问题的根本原因吗?
有没有人有任何建议,我可以尝试找到根本原因?

flvlnr44

flvlnr441#

该错误与Schema Registry的bootstrap.server配置有关,而不是Kafka本身,因为这需要_schemas__consumer_offsets主题存在,并且两者的默认复制因子都是3
如果你不需要Schema Registry,你可以忽略它,但是消费者客户端也不能读取没有__consumer_offsets主题的消费者组的数据,不管集群中有多少代理,或者你在其中设置了什么bootstrap.servers

相关问题