Kubernetes Kafka到Zookeeper-“Zookeeper客户端超时异常”错误

iklwldmw  于 2022-12-09  发布在  Apache
关注(0)|答案(1)|浏览(300)

作为上下文,我在Ubuntu机器上使用Kubernetes,通过Helm,在本地调出Kafka和Zookeeper:

- name: kafka
    version: 12.7.3
    repository: https://charts.bitnami.com/bitnami

我已经查看了这个错误的现有问题,但似乎没有一个与我的问题完全相关。对于这些现有的问题,我看到的问题似乎涉及到docker网络,或通信。然而,在我的本地设置,我可以看到Kafka * 可以 * 通信Zookeeper成功,并启动一个TCP连接。我看到以下tshark日志,其中.83是Kafka,.80是Zookeeper:

57 118.532604170 192.168.83.83 → 192.168.83.80 TCP 74 44978 → 2181 [SYN] Seq=0 Win=64800 Len=0 MSS=1440 SACK_PERM=1 TSval=3500466016 TSecr=0 WS=128
   58 118.532617080 192.168.83.80 → 192.168.83.83 TCP 74 2181 → 44978 [SYN, ACK] Seq=0 Ack=1 Win=64260 Len=0 MSS=1440 SACK_PERM=1 TSval=1996498322 TSecr=3500466016 WS=128
   59 118.532633329 192.168.83.83 → 192.168.83.80 TCP 66 44978 → 2181 [ACK] Seq=1 Ack=1 Win=64896 Len=0 TSval=3500466016 TSecr=1996498322
   60 118.535617526 192.168.83.83 → 192.168.83.80 TCP 115 44978 → 2181 [PSH, ACK] Seq=1 Ack=1 Win=64896 Len=49 TSval=3500466019 TSecr=1996498322
   61 118.535644624 192.168.83.80 → 192.168.83.83 TCP 66 2181 → 44978 [ACK] Seq=1 Ack=50 Win=64256 Len=0 TSval=1996498325 TSecr=3500466019
   62 118.537006985 192.168.83.80 → 192.168.83.83 TCP 107 2181 → 44978 [PSH, ACK] Seq=1 Ack=50 Win=64256 Len=41 TSval=1996498326 TSecr=3500466019
   63 118.537047974 192.168.83.83 → 192.168.83.80 TCP 66 44978 → 2181 [ACK] Seq=50 Ack=42 Win=64896 Len=0 TSval=3500466020 TSecr=1996498326
   64 118.540259005 192.168.83.83 → 192.168.83.80 TCP 78 44978 → 2181 [PSH, ACK] Seq=50 Ack=42 Win=64896 Len=12 TSval=3500466024 TSecr=1996498326
   65 118.540263332 192.168.83.80 → 192.168.83.83 TCP 66 2181 → 44978 [ACK] Seq=42 Ack=62 Win=64256 Len=0 TSval=1996498330 TSecr=3500466024
   66 118.541564514 192.168.83.80 → 192.168.83.83 SMPP 86 Bind_receiver[Malformed Packet]
   67 118.541607278 192.168.83.83 → 192.168.83.80 TCP 66 44978 → 2181 [ACK] Seq=62 Ack=62 Win=64896 Len=0 TSval=3500466025 TSecr=1996498331
   68 118.541999795 192.168.83.80 → 192.168.83.83 TCP 66 2181 → 44978 [FIN, ACK] Seq=62 Ack=62 Win=64256 Len=0 TSval=1996498331 TSecr=3500466025
   69 118.542214437 192.168.83.83 → 192.168.83.80 TCP 66 44978 → 2181 [FIN, ACK] Seq=62 Ack=63 Win=64896 Len=0 TSval=3500466026 TSecr=1996498331

尽管如此,我似乎仍然在Kafka日志中看到以下错误:

[2021-01-29 19:17:49,922] INFO Session: 0x1000031e07c0011 closed (org.apache.zookeeper.ZooKeeper)
[2021-01-29 19:17:49,922] INFO EventThread shut down for session: 0x1000031e07c0011 (org.apache.zookeeper.ClientCnxn)
[2021-01-29 19:17:49,925] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-01-29 19:17:49,928] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:262)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1881)
    at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:441)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:466)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
    at kafka.Kafka$.main(Kafka.scala:82)
    at kafka.Kafka.main(Kafka.scala)

我试过几种方法:
1.如上所述,我看到Kafka和Zookeeper之间的IP/TCP通信似乎确实工作成功,所以我不相信这是一个潜在的路由问题。
1.这是(1)所暗示的,但是我查看了nat表中的iptables规则,这些规则似乎是正确的。zookeeper服务被正确地NAT到zookeeper pod IP。
1.我试着在Kafka的pod中手动运行调试命令,再次确认它是否可以与Zookeeper建立端到端的连接。以下命令似乎有效:echo mntr | nc 10.96.85.98 2181 .
1.据我所知,我没有任何防火墙在运行。完全有可能iptables中有什么东西阻止了另一层的工作,但这是我希望得到一些澄清。

xzlaal3s

xzlaal3s1#

我现在可以使用这个功能了,这似乎是因为我反复地关闭和打开集群,并且没有正确地清除网络状态,这可能导致了某种黑洞。
这可能有点矫枉过正,但我最终做的只是简单地刷新iptables规则,并重新启动所有相关的服务,如docker,这需要特殊的iptables规则。

相关问题