canal 若select带有函数操作,会导致ES同步失败

hec6srdp  于 3个月前  发布在  其他
关注(0)|答案(1)|浏览(62)
  • I have searched the issues of this repository and believe that this is not a duplicate.
  • I have checked the FAQ of this repository and believe that this is not a duplicate.

environment

  • canal version 1.1.7 release
  • mysql version mysql 8.0
  • es version 7.17.13

Issue Description

mysql同步es,若sql select中带有函数操作,UPDATE or INSERT都会导致同步ES失败。

Steps to reproduce

select CAST(tr.trading_amt_sum*100 as UNSIGNED) as total_fee

Expected behaviour

Actual behaviour

If there is an exception, please attach the exception trace:

2024-02-27 09:11:33.132 [pool-3-thread-1] ERROR c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - java.util.NoSuchElementException
java.lang.RuntimeException: java.util.NoSuchElementException
    at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:112)
    at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:60)
    at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.sync(ESAdapter.java:104)
    at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.sync(ESAdapter.java:83)
    at com.alibaba.otter.canal.client.adapter.ProxyOuterAdapter.sync(ProxyOuterAdapter.java:42)
    at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.batchSync(AdapterProcessor.java:139)
    at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.lambda$null$1(AdapterProcessor.java:97)
    at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807)
    at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.lambda$null$2(AdapterProcessor.java:94)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.NoSuchElementException: null
    at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1000)
    at com.alibaba.otter.canal.client.adapter.es7x.support.ES7xTemplate.getESDataFromDmlData(ES7xTemplate.java:320)
    at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.singleTableSimpleFiledUpdate(ESSyncService.java:814)
    at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.update(ESSyncService.java:262)
    at com.alibaba.otter.canal.client.adapter.es.core.service.ESSyncService.sync(ESSyncService.java:97)
    ... 12 common frames omitted
2024-02-27 09:11:33.229 [Thread-2] ERROR c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - Outer adapter sync failed!  Error sync but ACK!

My Solution

client-adapter/escore/src/main/java/com/alibaba/otter/canal/client/adapter/es/core/service/ESSyncService.java

@@ -220,6 +220,10 @@ public class ESSyncService {
 
                     boolean allUpdateFieldSimple = true;
                     out: for (FieldItem fieldItem : schemaItem.getSelectFields().values()) {
+                        if (fieldItem.getColumnItems().isEmpty()) {
+                            allUpdateFieldSimple = false;
+                            break;
+                        }
                         for (ColumnItem columnItem : fieldItem.getColumnItems()) {
                             if (old.containsKey(columnItem.getColumnName())) {
sqougxex

sqougxex1#

我这边也碰到了这个情况,把函数去掉正常同步,加上函数后抛同样的异常并回滚,兄弟你这个问题解决了吗

相关问题