1) 我们有三节点Kafka和Kafka连接集群
2) 我们正在kafka节点上以分布式模式运行kafka connect
3) 当我尝试使用以下配置创建连接器时:
{
"name": "connector-state-0",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.user": "user",
"database.server.id": "5023",
"database.hostname": "hostname",
"database.password": "password",
"database.history.kafka.bootstrap.servers": "ip:9092",
"database.history.kafka.topic": "topicname",
"database.server.name": "prod",
"database.port": "3306",
"snapshot.mode": "when_needed",
"include.schema.changes": "false",
"table.whitelist": "country.state"
}
}
在请求创建连接器时,在3个节点中的2个节点上出现以下错误:
{"error_code":409,"message":"Cannot complete request because of a conflicting operation (e.g. worker rebalance)"}
在其中一个节点上:我可以创建连接器,但任务未启动,我可以在日志中看到以下错误:
[2019-01-23 10:50:06,455] INFO 127.0.0.1 - - [23/Jan/2019:10:50:06 +0000] "POST /connectors/birdeye-connector-state-0/tasks?forward=true HTTP/1.1" 409 113 8 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2019-01-23 10:50:06,462] INFO 127.0.0.1 - - [23/Jan/2019:10:50:06 +0000] "POST /connectors/birdeye-connector-state-0/tasks HTTP/1.1" 409 113 21 (org.apache.kafka.connect.runtime.rest.RestServer:60)
[2019-01-23 10:50:06,466] ERROR Request to leader to reconfigure connector tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1020)
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Cannot complete request because of a conflicting operation (e.g. worker rebalance)
at org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1017)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
我搞不清楚是什么引起了这场纠纷。
1条答案
按热度按时间oxcyiej71#
你需要设置
rest.advertised.host.name
到其他kafka connect工作程序可以解析并连接到的主机或ip。这是因为它用于员工之间的内部沟通。如果您的rest请求命中的工作线程不是集群的当前领导者,那么该工作线程将尝试将请求转发给领导者。它使用
rest.advertised.host.name
. 但如果rest.advertised.host.name
是localhost
然后,工人只需将请求转发给自己,因此事情就不起作用了。在你的三个工人中,有一个是领导者,这就是为什么你发现三分之二的人都失败了。有关详细信息,请参阅https://rmoff.net/2019/11/22/common-mistakes-made-when-configuring-multiple-kafka-connect-workers/