我在这里使用了debezium.io提供的教程,并将其调整为使用mssqlserver数据库而不是mysql,但是当我在数据库中进行更改时,watcher不会显示任何事件或活动。以下是我采取的步骤:
我运行了zookeeper docker命令: docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.1
然后我运行了kafka docker命令: docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.1
然后我运行了一个启用代理的sql server docker命令: docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -e 'MSSQL_AGENT_ENABLED=True' --name mssql -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-latest
然后我连接到sql server示例,创建了一个名为peopledb的数据库,并通过运行以下查询创建了一个名为people的表:
USE [PeopleDb] GO CREATE TABLE [dbo].[People]([Id] [bigint] IDENTITY(1,1) NOT NULL,[FirstName] [varchar](50) NOT NULL, [LastName] [varchar](50) NOT NULL, CONSTRAINT [PK_People] PRIMARY KEY CLUSTERED ( [Id] ASC)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]) ON [PRIMARY] GO
然后我运行以下命令来启用cdc:
Use PeopleDb Go EXEC sys.sp_cdc_enable_db
EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'People', @role_name = Null, @filegroup_name = N'Primary',@supports_net_changes = 0 EXEC sys.sp_cdc_help_change_data_capture
我注意到cdc表是在“系统表”下创建的。当我运行下面的查询时,一条记录被添加到cdc.dbo\u people\u ct表中: INSERT INTO TABLE dbo.People(FirstName, LastName) values ('John', 'Smith')
然后我运行connector docker命令: docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka --link mssql:mssql debezium/connect:1.1
然后我通过将下面的json发布到http://localhost:8083/连接器/:
{
"name": "people-connector",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"tasks.max": "1",
"database.hostname": "mssql",
"database.port": "1433",
"database.user": "sa",
"database.password": "yourStrong(!)Password",
"database.dbname": "PeopleDb",
"database.server.id": "184054",
"database.server.name": "mssql",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "mssql.dbo.people",
"name": "people-connector"
},
"tasks": [],
"type": "source"
}
我通过检查来验证新添加的连接器是否正在运行http://localhost:8083/连接器/人员连接器/状态: {"name":"people-connector","connector":{"state":"RUNNING","worker_id":"172.17.0.5:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"172.17.0.5:8083"}],"type":"source"}
我运行了watcher docker命令: docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.1 watch-topic -a -k mssql.dbo.people
结果如下:
WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations.
Using ZOOKEEPER_CONNECT=172.17.0.2:2181
Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.6:9092
Using KAFKA_BROKER=172.17.0.3:9092
Contents of topic mssql.dbo.people:
null {
"source" : {
"server" : "mssql"
},
"position" : {
"transaction_id" : null,
"event_serial_no" : 1,
"commit_lsn" : "00000025:000003f8:0003",
"change_lsn" : "NULL"
},
"databaseName" : "PeopleDb",
"schemaName" : "dbo",
"tableChanges" : [ {
"type" : "CREATE",
"id" : "\"PeopleDb\".\"dbo\".\"People\"",
"table" : {
"defaultCharsetName" : null,
"primaryKeyColumnNames" : [ "Id" ],
"columns" : [ {
"name" : "Id",
"jdbcType" : -5,
"typeName" : "bigint identity",
"typeExpression" : "bigint identity",
"charsetName" : null,
"length" : 19,
"scale" : 0,
"position" : 1,
"optional" : false,
"autoIncremented" : false,
"generated" : false
}, {
"name" : "FirstName",
"jdbcType" : 12,
"typeName" : "varchar",
"typeExpression" : "varchar",
"charsetName" : null,
"length" : 50,
"position" : 2,
"optional" : false,
"autoIncremented" : false,
"generated" : false
}, {
"name" : "LastName",
"jdbcType" : 12,
"typeName" : "varchar",
"typeExpression" : "varchar",
"charsetName" : null,
"length" : 50,
"position" : 3,
"optional" : false,
"autoIncremented" : false,
"generated" : false
} ]
}
} ]
}
完成这些步骤并确保有一个正在运行的连接器之后,我希望在将新记录插入people表或删除或更新它们时看到新的事件,但是观察者没有显示任何活动。有人知道为什么debezium和sqlserver之间似乎存在断开连接吗?
1条答案
按热度按时间jrcvhitl1#
根据debezium sql server connector文档:
sql server连接器将单个表上所有插入、更新和删除操作的事件写入单个kafka主题。kafka主题的名称总是采用servername.schemaname.tablename的形式,其中servername是用
database.server.name
配置属性,schemaname是发生操作的架构的名称,tablename是发生操作的数据库表的名称。在您的情况下,要查看
dbo.People
中的表mssql
你需要监视的数据库mssql.dbo.People
(请注意,Kafka中的主题名称区分大小写)。