以下命令用于在Docker中设置Zookeeper,Kafka,Schema-registry,Kafka-connect和MongoDb。Kakfa-connect集群似乎没有连接到MongoDb,它一直显示Connection refused
。所有容器都部署在一个名为Confluent的Docker网络中。
# create confluent network \
docker network create confluent && \
# create Zookeeper \
docker run -d \
--name=zookeeper \
--net=confluent \
-p 2181:2181 \
-e ZOOKEEPER_CLIENT_PORT=2181 \
confluentinc/cp-zookeeper:5.4.0 && \
# create kafka \
docker run -d \
--name=kafka \
--net=confluent \
-p 9092:9092 \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
confluentinc/cp-kafka:5.4.0 && \
# create schema-registry \
docker run -d \
--name=schema-registry \
--net=confluent \
-p 8081:8081 \
-e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2181 \
-e SCHEMA_REGISTRY_HOST_NAME=schema-registry \
-e SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081 \
confluentinc/cp-schema-registry:5.4.0 && \
# create kafka-connect \
docker run -d \
--name=kafka-connect \
--net=confluent \
-p 8083:8083 \
-e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
-e CONNECT_REST_PORT=8083 \
-e CONNECT_GROUP_ID="quickstart" \
-e CONNECT_CONFIG_STORAGE_TOPIC="quickstart-config" \
-e CONNECT_OFFSET_STORAGE_TOPIC="quickstart-offsets" \
-e CONNECT_STATUS_STORAGE_TOPIC="quickstart-status" \
-e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_REST_ADVERTISED_HOST_NAME="kafka-connect" \
-e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
-e CONNECT_PLUGIN_PATH=/usr/share/java \
-e CONNECT_REST_HOST_NAME="kafka-connect" \
-v /tmp/quickstart/file:/tmp/quickstart \
confluentinc/cp-kafka-connect:5.4.0 && \
# create mongo \
docker pull mongo && \
docker run -d --name=mongodb --net=confluent -v $(pwd)/data/db:/data/db -p 27017:27017 -d mongo --replSet debezium && \
docker exec -it mongodb mongosh --eval 'rs.initiate({_id: "debezium", members:[{_id: 0, host: "localhost:27017"}]})'
我使用以下命令来设置Kafka Debezium MongoDb连接器
docker exec -it kafka-connect bash
confluent-hub install debezium/debezium-connector-mongodb:0.9.4
docker restart kafka-connect
最后,我在mongosh中使用以下代码为Debezium创建用户
use admin
db.createUser(
{
user: "debezium",
pwd: "dbz",
roles: ["dbOwner"]
})
现在我尝试使用以下json文件设置MongoDb Debezium Kafka连接器
{
"name": "inventory-connector",
"config": {
"connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
"tasks.max": "1",
"mongodb.hosts": "debezium/127.0.0.1:27017",
"mongodb.name": "dbserver1",
"mongodb.user": "debezium",
"mongodb.password": "dbz"
}
}
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8083/connectors/ -d @register-mongodb.json
现在,当我触发上面的curl命令时,下面是我在Kafka-connect docker容器中看到的错误消息
[2023-04-29 12:58:26,106] DEBUG Getting plugin class loader for connector: 'io.debezium.connector.mongodb.MongoDbConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
connector.class = io.debezium.connector.mongodb.MongoDbConnector
connector.class = io.debezium.connector.mongodb.MongoDbConnector
[2023-04-29 12:58:26,132] DEBUG Getting plugin class loader for connector: 'io.debezium.connector.mongodb.MongoDbConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2023-04-29 12:58:26,144] INFO Cluster description not yet available. Waiting for 30000 ms before timing out (org.mongodb.driver.cluster)
[2023-04-29 12:58:43,303] DEBUG Closing connection connectionId{localValue:36} (org.mongodb.driver.connection)
[2023-04-29 12:58:43,304] DEBUG Updating cluster description to
{type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception=
{com.mongodb.MongoSocketOpenException: Exception opening socket},
caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
(org.mongodb.driver.cluster)
Kafka-connect集群似乎无法连接到MongoDb。你知道这是怎么回事吗
1条答案
按热度按时间jhdbpxl91#
如果使用斜杠,则表示这不是有效的主机名,并且没有一个容器名为debezium。你想连接到Mongo容器。
设置
--name=mongodv
。使用它,而不是像错误所说的那样,试图让Connect容器启动一个指向自己的地址的Mongo客户端。