无法使用apachekafka和debezium从联合mysql表读取数据

oipij1gg  于 2021-06-04  发布在  Kafka
关注(0)|答案(1)|浏览(378)

我使用apachekafka和debezium的连接器服务来查看表中的数据更改。我还使用了awsec2,其中有一个带有联邦表的数据库。如果我创建了一个无联邦表,并使用zookeeper、kafka和kafka connect up运行docker watcher,我可以从该表中读取数据,但是当我看到一个联邦表时,就不能读取数据。
我试图在连接器的配置中添加一个查询,但这不起作用。
我的命令是:

启动zookeeper $ docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:0.9 ####开始Kafka $ docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:0.9 ####启动Kafka连接 $ docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka debezium/connect:0.9 ####注册连接器 $ curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{ "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "tasks.max": "1", "database.hostname": "mysql", "database.port": "3306", "database.user": "debezium", "database.password": "dbz", "database.server.id": "184054", "database.server.name": "dbserver1", "database.whitelist": "inventory", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "dbhistory.inventory" } }' ####显示数据更改 $ docker run -it --name watcher --rm --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:0.9 watch-topic -a -k dbserver1.inventory.customers 我希望看到这样的情况:

null    {"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":true,"field":"a1"},{"type":"string","optional":true,"field":"a2"}],"optional":true,"name":"dbserver1.cobranza.foobar_1.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":true,"field":"a1"},{"type":"string","optional":true,"field":"a2"}],"optional":true,"name":"dbserver1.cobranza.foobar_1.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":true,"field":"version"},{"type":"string","optional":true,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"server_id"},{"type":"int64","optional":false,"field":"ts_sec"},
...
txu3uszq

txu3uszq1#

在看 FEDERATED 发动机说明https://dev.mysql.com/doc/refman/8.0/en/federated-storage-engine.html 在我看来,对联邦表所做的更改并没有记录在服务器binlog文件中。如果是这样的话,那么debezium就不能流化这些变化,因为它看不到它们。
它应该能够从源表流式传输更改。

相关问题