canal mysql同步数据至ES数据丢失

myss37ts  于 4个月前  发布在  Mysql
关注(0)|答案(2)|浏览(169)

版本信息

  • mysql: 8.0.12
  • elasticsearch: 7.12.0
  • canal-admin: 1.1.6
  • acnal-server: 1.1.6
  • canal-adapter: 1.1.6

现象描述:
mysql为主从架构,canal同步mysql至ES时,adapter正常打印日志,es里面没有数据。mysql从库是有数据的,但是ES里面没有。数据量为1000+W,丢失数据量5条左右。以下为正常打印的日志信息,但是该日志信息的数据没有同步到ES中,只单独提取一条数据的日志信息,数据ID为:901065516406276100

2023-08-23 11:12:40.285 [pool-10-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":901065516406276100,"status":40,"type":10,"collect_time":1692760311000,"c_line_id":"1acd37010a5c11ee8d8ca24ec3edcd76","b_line_id":null,"c_class_id":null,"b_class_id":null,"c_task_id":"cf6a767d414511ee8c6ecc827f384eb2","b_task_id":null,"brand_id":null,"oper_user_id":null,"cgt_code":"AC019102","box_bar_team_dept_code":"123401010601","bar_piece_team_dept_code":null,"bar_piece_user_name":null,"group_date":null,"check_box_status":null,"scrap_status":null,"batch_code":null,"box_bar_batch_code":null,"bar_piece_batch_code":null}],"database":"tobacco_cloud_qrcode_sys","destination":"mysql2","es":1692760360000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"qr_code_ins103","ts":1692760360285,"type":"INSERT"}

2023-08-23 11:12:40.289 [pool-10-thread-1] DEBUG c.a.o.canal.client.adapter.es.core.service.ESSyncService - DML: {"data":[{"id":901065516406276100,"status":40,"type":10,"collect_time":1692760311000,"c_line_id":"1acd37010a5c11ee8d8ca24ec3edcd76","b_line_id":null,"c_class_id":null,"b_class_id":null,"c_task_id":"cf6a767d414511ee8c6ecc827f384eb2","b_task_id":null,"brand_id":null,"oper_user_id":null,"cgt_code":"AC019102","box_bar_team_dept_code":"123401010601","bar_piece_team_dept_code":null,"bar_piece_user_name":null,"group_date":null,"check_box_status":null,"scrap_status":null,"batch_code":null,"box_bar_batch_code":null,"bar_piece_batch_code":null}],"database":"tobacco_cloud_qrcode_sys","destination":"mysql2","es":1692760360000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"qr_code_ins103","ts":1692760360285,"type":"INSERT"}
Affected indexes: qrcode_ins_prod

siotufzp

siotufzp1#

同问,数据量只有几百条,由于是测试机器。因此网络有延迟。但是丢失数据没有征兆,只能考虑使用rabbitmq来接收了,,,

r8xiu3jd

r8xiu3jd2#

我也是有这个问题,大概千分之1的到百分之1的之间,

相关问题