elasticsearch中的uax_url_email tokenizer为带有特殊字符的电子邮件生成多个令牌

aij0ehis  于 2023-01-08  发布在  ElasticSearch
关注(0)|答案(1)|浏览(90)

我使用uax_url_email tokenizer为我们的索引中的电子邮件字段。它工作完美,并生成像johndoe@yahoo.com的普通电子邮件的单一令牌。但是,它生成多个令牌时,电子邮件有外国或特殊字符。有解决这个问题的办法吗?我不希望生成多个令牌

PUT email-test-index
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "email_analyzer": {
            "filter": ["lowercase"],
            "tokenizer": "email_tokenizer"
          }
        },
        "tokenizer": {
          "email_tokenizer": {
            "type": "uax_url_email"
          }
        }
      }
    }
  },
  "mappings": {
    "date_detection": false,
    "numeric_detection": false,
    "properties": {
      "EMAIL": {
        "type": "text",
        "store": true,
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        },
        "analyzer": "email_analyzer"
      }
    }
  }
}

当它工作时:

GET email-test-index/_analyze
{
  "field": "EMAIL",
  "text": "johndoe@yahoo.com"
}

{
  "tokens" : [
    {
      "token" : "johndoe@yahoo.com",
      "start_offset" : 0,
      "end_offset" : 17,
      "type" : "<EMAIL>",
      "position" : 0
    }
  ]
}

当它不工作时:

GET email-test-index/_analyze
{
  "field": "EMAIL",
  "text": "johndoeó8@yahoo.com"
}

{
  "tokens" : [
    {
      "token" : "johndoeó8",
      "start_offset" : 0,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "yahoo.com",
      "start_offset" : 10,
      "end_offset" : 19,
      "type" : "<URL>",
      "position" : 1
    }
  ]
}
qkf9rpyu

qkf9rpyu1#

尾巴;

你不能没有摆脱特殊字符。我可能是错的,但我不认为这样的字符甚至是允许的电子邮件标准。

溶液

您可以使用Map字符过滤器并捕获所有非ascii字符,以将它们Map到ascii。

POST _analyze
{
  "tokenizer": "uax_url_email",
  "char_filter": [
    {
      "type": "mapping",
      "mappings": [
        "ó => o"
      ]
    }
  ],
  "text": "Email me at johndóe8@yahoo.com"
}

{
  "tokens": [
    {
      "token": "Email",
      "start_offset": 0,
      "end_offset": 5,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "me",
      "start_offset": 6,
      "end_offset": 8,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "at",
      "start_offset": 9,
      "end_offset": 11,
      "type": "<ALPHANUM>",
      "position": 2
    },
    {
      "token": "johndoe8@yahoo.com",
      "start_offset": 12,
      "end_offset": 30,
      "type": "<EMAIL>",
      "position": 3
    }
  ]
}

请注意,ó已替换为"o"

相关问题