Question
canal-adapter 日志中时不时 document missing的异常,2021-04-13 14:41:40.633 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][4]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114902]: document missing]]
但是每次只会打印一两次,而且去Es查询也可以查询到这个数据。
但是在运行了一天后,adapter日志开始疯狂打印这个日志,并且这条数据在Elasticsearch中也查询不到。
2021-04-13 14:42:36.770 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][3]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114875]: document missing]]
2021-04-13 14:42:36.770 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][3]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114881]: document missing]]
2021-04-13 14:42:36.770 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][4]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114902]: document missing]]
2021-04-13 14:42:36.770 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][1]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114904]: document missing]]
2021-04-13 14:42:36.770 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][4]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114905]: document missing]]
2021-04-13 14:42:36.770 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][1]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114906]: document missing]]
2021-04-13 14:42:37.361 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][3]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114875]: document missing]]
2021-04-13 14:42:37.361 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][3]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114881]: document missing]]
2021-04-13 14:42:37.361 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][4]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114902]: document missing]]
2021-04-13 14:42:37.361 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][1]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114904]: document missing]]
2021-04-13 14:42:37.361 [pool-2-thread-1] ERROR c.a.otter.canal.client.adapter.es7x.support.ESConnection - [order_form_index_1/pHcBvUSJTvuw3dFrbpQwJQ][[order_form_index_1][4]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][114905]: document missing]]
目前有三个示例文件但是只有一个是父子文档,只有父子文档的示例会出现这个异常,其他两个未出现任何异常
请问有没有人遇到过这个问题或者谁能提供一点思路,着急上生产了。
下面附上我的配置文件:前者为父文档,后者为子文档
dataSourceKey: ecDS
outerAdapterKey: order_form
destination: mes_order_form
groupId: g1
esMapping:
_index: order_form_index_1
_id: id
_type: _doc
relations:
order_form:
name: order
sql: "SELECT
GROUP_CONCAT('{\"id\":',pta.id,',\"title\":\"',pta.alias,'\",\"value\":\"',foa.replenish_value,'\",\"orderListViewFlag\":',(case when foa.order_list_view_flag = 1 then 'true' else 'false' end),'}') as info,
a.id AS id,
a.change_flag,
a.delete_flag,
a.delivery_date AS delivery_time,
a.discount_type,
a.max_out_stock_time as out_stock_time,
a.code_init_time,
a.status,
a.code,
a.order_date,
a.pay_attributes,
a.pay_discount,
a.pay_price,
a.pay_reduce,
a.remark,
a.start_date AS start_time,
a.unit_price,
a.amount AS work_count,
a.relate_order_id,
a.product_title,
a.product_title AS product_name,
s.sale_store AS store,
u.real_name AS middleman,
b.org_code,
c.short_name AS client,
c.middleman_group_id,
p.product_type_id,
m.title,
fo.all_search_value AS info_search,
a.sale_id,
a.company_id,
a.client_id,
a.product_id,
s.middleman_id,
a.calculation_method_id,
foa.product_attribute_id,
pa.product_type_attribute_id
FROM
form_order a
LEFT JOIN form_sale s ON a.sale_id = s.id
LEFT JOIN company b ON a.company_id = b.id
LEFT JOIN client c ON a.client_id = c.id
LEFT JOIN product p ON a.product_id = p.id
LEFT JOIN `user` u ON s.middleman_id = u.id
LEFT JOIN calculation_method m ON a.calculation_method_id = m.id
LEFT JOIN form_order_search fo ON a.id = fo.order_id
LEFT JOIN form_order_attribute foa ON a.id = foa.order_id
LEFT JOIN product_attribute pa ON foa.product_attribute_id = pa.id
LEFT JOIN product_type_attribute pta ON pa.product_type_attribute_id = pta.id
GROUP BY a.id"
dataSourceKey: fpmsDS
outerAdapterKey: order_form
destination: mes_order_form
groupId: g1
esMapping:
_index: order_form_index_1
_id: id
_type: _doc
relations:
order_form:
name: form
parent: union_code
sql: "SELECT
concat('form_', w.id) as id,
o.union_code,
CONCAT(w.work_code,(case when w.suffix = NULL then '' else w.suffix end)) as work_code,
w.order_id,
w.revoke_flag,
w.company_id
FROM
work_form w
LEFT JOIN order_base o ON o.id = w.order_id"
commitBatch: 3000
5条答案
按热度按时间t9aqgxwy1#
我也遇到这个问题,应该是insert的时候,操作被放入了缓冲区,es没有及时提交,但是update是即时提交的,导致出现document missing
zsbz8rwp2#
我也遇到这个问题,应该是insert的时候,操作被放入了缓冲区,es没有及时提交,但是update是即时提交的,导致出现document missing
谢谢回答,正常情况下报错也是可以插入成功的,但是运行一天后突然一条数据疯狂打印documentmissing信息,反查ES发现疯狂刷日志这条数据没有成功插入到es中,您知道为什么会有数据插入失败吗
nfs0ujit3#
你好 请问解决了吗 我也遇到了这个问题
ekqde3dh4#
我也遇到这个问题,应该是insert的时候,操作被放入了缓冲区,es没有及时提交,但是update是即时提交的,导致出现document missing
谢谢回答,正常情况下报错也是可以插入成功的,但是运行一天后突然一条数据疯狂打印documentmissing信息,反查ES发现疯狂刷日志这条数据没有成功插入到es中,您知道为什么会有数据插入失败吗
请问这个问题解决了吗
vpfxa7rd5#
请问这个问题解决了吗