使用多个Kafka代理和node-rdkafka admin createTopic的复制给出超时

2vuwiymt  于 2023-08-02  发布在  Apache
关注(0)|答案(1)|浏览(114)

我试图建立一个环境来学习Kafka的复制。但是,执行下面的代码会出现超时错误。我找不出问题出在哪里。
如果我使用带有Docker容器的单个代理示例,并且没有复制,它会工作得很好。

error creating topic:  LibrdKafkaError: Local: Timed out
        at Function.createLibrdkafkaError [as create] (/Users/my-name/Developer/test/kafka-nodejs/kafka-local-docker/node_modules/node-rdkafka/lib/error.js:456:10)
        at /Users/my-name/Developer/test/kafka-nodejs/kafka-local-docker/node_modules/node-rdkafka/lib/admin.js:132:28 {
      code: -185,
      errno: -185,
      origin: 'kafka'
    }
const Kafka = require('node-rdkafka');
    
    const admin = Kafka.AdminClient.create({
        'bootstrap.servers': 'localhost:9091,localhost:9092,localhost:9093',
        'broker.version.fallback': '0.10.2.1',
        'log.connection.close' : false
    });
    
    const topicName = 'cool-topic-4';
    
    const newTopic = {
        topic: topicName,
        num_partitions: 2,
        replication_factor: 3,
        config: {
            'min.insync.replicas': '2',
        }
    }
    
    // create a new kafka topic 
    admin.createTopic(newTopic, (err) => {
            if (err)
                console.log("error creating topic: ", err);
            else
                console.log("topic created: ", topicName);
        }
    );
---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    hostname: zookeeper
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka1:
    image: confluentinc/cp-kafka:7.3.0
    container_name: kafka1
    ports:
      - "9091:9091"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_NODE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9091,PLAINTEXT_NODE://kafka1:29091

  
  kafka2:
    image: confluentinc/cp-kafka:7.3.0
    container_name: kafka2
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_NODE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_NODE://kafka2:29092

  
  kafka3:
    image: confluentinc/cp-kafka:7.3.0
    container_name: kafka3
    ports:
      - "9093:9093"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_NODE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9093,PLAINTEXT_NODE://kafka3:29093

不知道这是否有帮助,但这里是docker container logs的一部分:

kafka2     |    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
kafka2     | [2023-07-28 15:56:38,308] INFO [Controller id=2, targetBrokerId=1] Client requested connection close from node 1 (org.apache.kafka.clients.NetworkClient)
kafka2     | [2023-07-28 15:56:38,410] INFO [Controller id=2, targetBrokerId=1] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka2     | [2023-07-28 15:56:38,410] WARN [Controller id=2, targetBrokerId=1] Connection to node 1 (localhost/127.0.0.1:9091) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka2     | [2023-07-28 15:56:38,410] INFO [Controller id=2, targetBrokerId=3] Node 3 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka2     | [2023-07-28 15:56:38,410] WARN [Controller id=2, targetBrokerId=3] Connection to node 3 (localhost/127.0.0.1:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka2     | [2023-07-28 15:56:38,410] WARN [RequestSendThread controllerId=2] Controller 2's connection to broker localhost:9093 (id: 3 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
kafka2     | java.io.IOException: Connection to localhost:9093 (id: 3 rack: null) failed.
kafka2     |    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)
kafka2     |    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:292)
kafka2     |    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:246)
kafka2     |    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
kafka2     | [2023-07-28 15:56:38,410] INFO [Controller id=2, targetBrokerId=3] Client requested connection close from node 3 (org.apache.kafka.clients.NetworkClient)
kafka2     | [2023-07-28 15:56:38,410] WARN [RequestSendThread controllerId=2] Controller 2's connection to broker localhost:9091 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
kafka2     | java.io.IOException: Connection to localhost:9091 (id: 1 rack: null) failed.
kafka2     |    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)

0x6upsns

0x6upsns1#

您需要正确设置代理间协议。默认情况下,它是PLAINTEXT,但是您已经将通告的侦听器设置为localhost而不是容器服务名称,因此每个代理将尝试在不正确的端口上连接到自己

相关问题