由于数据类型不匹配,kinesis firehose无法将数据传递到elasticsearch

enyaitl3  于 2021-06-10  发布在  ElasticSearch
关注(0)|答案(0)|浏览(298)

我有一个kinesis交付流设置,它从kinesis数据流读取json数据(解析的binlog cdc数据),并将其交付给elasticsearch(aws管理)。但是,很少有记录因为数据类型不匹配而失败。

{
  "attemptsMade": 8,
  "arrivalTimestamp": 1601040754403,
  "errorCode": "400",
  "errorMessage": "{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [data.entity_id] of type [long] in document with id \\u002749611071821873828133613420294125495454071877611827494914.2\\u0027. Preview of field\\u0027s value: \\u0027TMF0003210\\u0027\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"TMF0003210\\\"\"}}",
  "attemptEndingTimestamp": 1601041110351,
  "rawData": "eyJkYXRhYmFzZSI6ImNtc191YXQiLCJ0YWJsZSI6InZwX2pvYiIsInR5cGUiOiJ1cGRhdGUiLCJ0cyI6MTYwMTA0MDc1NCwieGlkIjo0Mzc2NTk0OSwiY29tbWl0Ijp0cnVlLCJwb3NpdGlvbiI6Im15c3FsLWJpbi1jaGFuZ2Vsb2cuMDAwMDc0Ojk0NjI0NjE1Iiwic2VydmVyX2lkIjoxNzY0MTQ4MjYxLCJ0aHJlYWRfaWQiOjQ1ODI0Mywic2NoZW1hX2lkIjo1NiwiZGF0YSI6eyJpZCI6NzQwMTksImVudGl0eV9pZCI6IlRNRjAwMDMyMTAiLCJqb2JfaWQiOiJmY2U2MDVhMS1kZmQxLTRkNDgtODdlOC0xNjRhODljZWI0ZGYiLCJzdGF0dXMiOiJDTVNfVVBEQVRFRCIsInN0YXJ0X3RpbWUiOiIyMDIwLTA5LTI1IDEzOjMyOjMzIiwiZW5kX3RpbWUiOiIyMDIwLTA5LTI1IDEzOjMyOjM0In0sIm9sZCI6eyJzdGF0dXMiOiJTVEFSVEVEIiwiZW5kX3RpbWUiOm51bGx9fQ==",
  "subsequenceNumber": 2,
  "esDocumentId": "49611071821873828133613420294125495454071877611827494914.2",
  "esIndexName": "cdc-nonprod-2020-09-25",
  "esTypeName": ""
}

字段实体\u id在某些表中被定义为整数,并且只包含数字数据,但是很少有其他表具有varchar(10)类型,并且它们包含相同字段名的文本数据。因此elasticsearch以某种方式假设它是一个数字字段,并且在接收同名字段的字符串值时出错。
如何处理此错误?是否可以为同一字段名设置不同的数据类型?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题