Kafka Debezium Mongo源连接器无法连接到MongoDb示例

moiiocjp  于 2023-05-06  发布在  Apache
关注(0)|答案(1)|浏览(183)

以下命令用于在Docker中设置Zookeeper,Kafka,Schema-registry,Kafka-connect和MongoDb。Kakfa-connect集群似乎没有连接到MongoDb,它一直显示Connection refused。所有容器都部署在一个名为Confluent的Docker网络中。

# create confluent network \ 
docker network create confluent && \

# create Zookeeper \ 
docker run -d \
    --name=zookeeper \
    --net=confluent \
    -p 2181:2181 \
    -e ZOOKEEPER_CLIENT_PORT=2181 \
    confluentinc/cp-zookeeper:5.4.0 && \ 

# create kafka \ 
docker run -d \
    --name=kafka \
    --net=confluent \
    -p 9092:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka:5.4.0 && \

# create schema-registry \ 
docker run -d \
  --name=schema-registry \
  --net=confluent \
  -p 8081:8081 \
  -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2181 \
  -e SCHEMA_REGISTRY_HOST_NAME=schema-registry \
  -e SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081 \
  confluentinc/cp-schema-registry:5.4.0 && \

# create kafka-connect \ 
docker run -d \
  --name=kafka-connect \
  --net=confluent \
  -p 8083:8083 \
  -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
  -e CONNECT_REST_PORT=8083 \
  -e CONNECT_GROUP_ID="quickstart" \
  -e CONNECT_CONFIG_STORAGE_TOPIC="quickstart-config" \
  -e CONNECT_OFFSET_STORAGE_TOPIC="quickstart-offsets" \
  -e CONNECT_STATUS_STORAGE_TOPIC="quickstart-status" \
  -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
  -e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
  -e CONNECT_REST_ADVERTISED_HOST_NAME="kafka-connect" \
  -e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
  -e CONNECT_PLUGIN_PATH=/usr/share/java \
  -e CONNECT_REST_HOST_NAME="kafka-connect" \
  -v /tmp/quickstart/file:/tmp/quickstart \
  confluentinc/cp-kafka-connect:5.4.0 && \

# create mongo \ 
docker pull mongo && \
docker run -d --name=mongodb --net=confluent  -v $(pwd)/data/db:/data/db -p 27017:27017 -d mongo  --replSet debezium && \
docker exec -it mongodb mongosh --eval 'rs.initiate({_id: "debezium", members:[{_id: 0, host: "localhost:27017"}]})'

我使用以下命令来设置Kafka Debezium MongoDb连接器

docker exec -it kafka-connect bash
confluent-hub install debezium/debezium-connector-mongodb:0.9.4
docker restart kafka-connect

最后,我在mongosh中使用以下代码为Debezium创建用户

use admin
db.createUser(
 {
  user: "debezium",
  pwd: "dbz",
  roles: ["dbOwner"]
 })

现在我尝试使用以下json文件设置MongoDb Debezium Kafka连接器

{
    "name": "inventory-connector",
    "config": {
        "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
        "tasks.max": "1",
        "mongodb.hosts": "debezium/127.0.0.1:27017",
        "mongodb.name": "dbserver1",
        "mongodb.user": "debezium",
        "mongodb.password": "dbz"
    }
}

curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" http://localhost:8083/connectors/ -d @register-mongodb.json

现在,当我触发上面的curl命令时,下面是我在Kafka-connect docker容器中看到的错误消息

[2023-04-29 12:58:26,106] DEBUG Getting plugin class loader for connector: 'io.debezium.connector.mongodb.MongoDbConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
    connector.class = io.debezium.connector.mongodb.MongoDbConnector
    connector.class = io.debezium.connector.mongodb.MongoDbConnector
[2023-04-29 12:58:26,132] DEBUG Getting plugin class loader for connector: 'io.debezium.connector.mongodb.MongoDbConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2023-04-29 12:58:26,144] INFO Cluster description not yet available. Waiting for 30000 ms before timing out (org.mongodb.driver.cluster)

[2023-04-29 12:58:43,303] DEBUG Closing connection connectionId{localValue:36} (org.mongodb.driver.connection)
[2023-04-29 12:58:43,304] DEBUG Updating cluster description to  
{type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception=
{com.mongodb.MongoSocketOpenException: Exception opening socket}, 
caused by {java.net.ConnectException: Connection refused (Connection refused)}}] 
(org.mongodb.driver.cluster)

Kafka-connect集群似乎无法连接到MongoDb。你知道这是怎么回事吗

jhdbpxl9

jhdbpxl91#

debezium/127.0.0.1:27017

如果使用斜杠,则表示这不是有效的主机名,并且没有一个容器名为debezium。你想连接到Mongo容器。
设置--name=mongodv。使用它,而不是像错误所说的那样,试图让Connect容器启动一个指向自己的地址的Mongo客户端。

相关问题