kakfa connect的debezium sqlserverconnector未在kafka中创建主题

acruukt9  于 2021-06-06  发布在  Kafka
关注(0)|答案(1)|浏览(344)

我正在尝试从debezium获得sqlserverconnector，它在kafka connect中工作，除了没有创建任何主题外，其他一切似乎都运行良好。我错过什么了吗？
它以正确的配置开始任务。。。

Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,424] INFO Kafka version : 2.1.1-cp1 (org.apache.kafka.common.utils.AppInfoParser:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,424] INFO Kafka commitId : f5b753880d5460f1 (org.apache.kafka.common.utils.AppInfoParser:110)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,425] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:860)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO Starting SqlServerConnectorTask with configuration: (io.debezium.connector.common.BaseSourceTask:42)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    connector.class = io.debezium.connector.sqlserver.SqlServerConnector (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    database.user = cdc_test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    database.dbname = kafka_sql_test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    task.class = io.debezium.connector.sqlserver.SqlServerConnectorTask (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.hostname = mbvmsqlsvr001-t (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.history.kafka.bootstrap.servers = horton1:9092,horton2:9092,horton3:9092,horton4:9092 (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.history.kafka.topic = dbhistory.sql2016test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.password =********(io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    name = sql-server-test-connector (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.server.name = mbvmsqlsvr001-t (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.port = 1433 (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    table.whitelist = dbo.message_table (io.debezium.connector.common.BaseSourceTask:44)

获取快照设置。。。。。。

Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,430] INFO Kafka version : 2.1.1-cp1 (org.apache.kafka.common.utils.AppInfoParser:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,430] INFO Kafka commitId : f5b753880d5460f1 (org.apache.kafka.common.utils.AppInfoParser:110)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,522] INFO Requested thread factory for connector SqlServerConnector, id = mbvmsqlsvr001-t named = change-event-source-coordinator (io.debezium.util.Threads:23
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO Creating thread debezium-sqlserverconnector-mbvmsqlsvr001-t-change-event-source-coordinator (io.debezium.util.Threads:247)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO WorkerSourceTask{id=sql-server-test-connector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:19
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO No previous offset has been found (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource:56)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO According to the connector configuration both schema and data will be snapshotted (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSourc
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO Snapshot step 1 - Preparing (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:85)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,524] INFO Snapshot step 2 - Determining captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:94)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,528] INFO Snapshot step 3 - Locking captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:100)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Schema locking was disabled in connector configuration (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource:84)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 4 - Determining snapshot offset (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:106)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 5 - Reading structure of captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 6 - Persisting schema history (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:113)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,531] INFO Snapshot step 7 - Snapshotting data (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:125)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,531] INFO Snapshot step 8 - Finalizing (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:149)

连接器任务正在运行。。。


# curl  http://172.24.32.10:8083/connectors/sql-server-test-connector/status | jq

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   175  100   175    0     0  73160      0 --:--:-- --:--:-- --:--:-- 87500
{
  "name": "sql-server-test-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "horton5:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "horton5:8083"
    }
  ],
  "type": "source"
}

用户名和密码在sql server上有一个spid，并运行cdc查询以获取更改，但在kafka主题中没有显示任何内容。
我不明白什么？我是否应该先创建主题，然后如何配置连接器以向其发布消息？
更新：这里是连接器json

curl -X POST -H "Content-Type: application/json" --data '{
  "name": "sql-server-test-connector",
  "config": {
    "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
    "database.hostname": "mbvmsqlsvr001-t",
    "database.port": "1433",
    "database.user": "cdc_test",
    "database.password": "Password!123",
    "database.dbname": "kafka_sql_test",
    "database.server.name": "mbvmsqlsvr001-t",
    "table.whitelist": "dbo.message_table",
    "database.history.kafka.bootstrap.servers": "horton1:9092,horton2:9092,horton3:9092,horton4:9092",
    "database.history.kafka.topic": "dbhistory.sql2016test"
    }
  }' http://172.24.32.10:8083/connectors
apache-kafkasql-serverapache-kafka-connectdebezium

1条答案

按热度按时间
lf5gs5x2

lf5gs5x21#

固定的。
我自己的愚蠢错误。我使用的是debezium-connector-sqlserver-0.9.0.alpha1-plugin.tar.gz发行版。我把debezium-connector-sqlserver-0.9.4.final-plugin.tar.gz放在适当的位置，一切都很完美（我几乎可以实时看到消费者的变化）。
我甚至不知道我是怎么得到阿尔法版本的。。。。。。。

赞(0）分享  回复(0）举报  2021-06-06
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com