我正在使用标准的汇合jdbc连接器和microsoft的jdbc驱动程序连接到microsoft sql server数据库。对于一张特殊的table mode=timestamp
给它一个timestamp列或者我把它设置为 mode=bulk
,我只得到100个结果。从来没有比这更多的流。下面是我连接日志的结尾一直在说的:
[2018-09-19 12:51:51,393] INFO WorkerSourceTask{id=MES-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:199)
[2018-09-19 12:51:51,412] INFO After filtering the tables are: "MES2ERP"."dbo"."BatchDetails" (io.confluent.connect.jdbc.source.TableMonitorThread:175)
[2018-09-19 12:51:51,412] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:868)
[2018-09-19 12:51:51,475] INFO Cluster ID: GMY7vibUSHGp5xRekGDEwg (org.apache.kafka.clients.Metadata:285)
[2018-09-19 12:52:51,069] INFO WorkerSourceTask{id=MES-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:397)
[2018-09-19 12:52:51,070] INFO WorkerSourceTask{id=MES-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:414)
[2018-09-19 12:52:51,077] INFO WorkerSourceTask{id=MES-0} Finished commitOffsets successfully in 7 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:496)
[2018-09-19 12:53:51,082] INFO WorkerSourceTask{id=MES-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:397)
[2018-09-19 12:53:51,082] INFO WorkerSourceTask{id=MES-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:414)
我意识到 batch.max.rows
默认值是100,但是 mode=timestamp
,我应该让表中的所有行都流化,而不是在100之后停止。我希望 bulk
最终也会流式传输所有内容,或者至少每100个重复一次 table.poll.interval.ms
,但它不。。。就这么停了。知道怎么回事吗?
哦,我的table上有成百上千行。
这是我的连接器配置
name=JSON_OCN_MES-BatchDetails
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
key.converter.schemas.enable=false
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
connection.url=jdbc:sqlserver://####;databaseName=####;user=####;password=####;
table.whitelist=BatchDetails
mode=bulk
validate.non.null=false
topic.prefix=JSON_OCN_MES-
1条答案
按热度按时间aiqt4smr1#
您可以尝试更改连接器中的默认(100)batch.max.rows设置