elasticsearch Elastic Search重新索引以静默方式失败

jfewjypa  于 2023-04-11  发布在  ElasticSearch
关注(0)|答案(1)|浏览(129)

我在ElasticSearch中尝试语义搜索,遵循这个tutorial
当我复制一个索引文件到另一个索引[重新索引]以下命令

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "collection"
  },
  "dest": {
    "index": "collection-with-embeddings",
    "pipeline": "text-embeddings"
  }
}

有些文件在新索引中丢失了。但我不知道原因。我正在努力找出原因。
关于上下文,

PUT _ingest/pipeline/text-embeddings
{
  "description": "Text embedding pipeline",
  "processors": [
    {
      "inference": {
        "model_id": "sentence-transformers__msmarco-minilm-l-12-v3",
        "target_field": "text_embedding",
        "field_map": {
          "text": "text_field"
        }
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "description": "Index document to 'failed-<index>'",
        "field": "_index",
        "value": "failed-{{{_index}}}"
      }
    },
    {
      "set": {
        "description": "Set error message",
        "field": "ingest.failure",
        "value": "{{_ingest.on_failure_message}}"
      }
    }
  ]
}

这是任务详细信息

{
    "completed": true,
    "task": {
        "node": "YgR8udaSSMqClwCGWOBGBw",
        "id": 5946104,
        "type": "transport",
        "action": "indices:data/write/reindex",
        "status": {
            "total": 2414,
            "updated": 1346,
            "created": 1068,
            "deleted": 0,
            "batches": 3,
            "version_conflicts": 0,
            "noops": 0,
            "retries": {
                "bulk": 0,
                "search": 0
            },
            "throttled_millis": 0,
            "requests_per_second": -1.0,
            "throttled_until_millis": 0
        },
        "description": "reindex from [source_index] to [destination_index]",
        "start_time_in_millis": 1680795982705,
        "running_time_in_nanos": 22702121635,
        "cancellable": true,
        "cancelled": false,
        "headers": {}
    },
    "response": {
        "took": 22699,
        "timed_out": false,
        "total": 2414,
        "updated": 1346,
        "created": 1068,
        "deleted": 0,
        "batches": 3,
        "version_conflicts": 0,
        "noops": 0,
        "retries": {
            "bulk": 0,
            "search": 0
        },
        "throttled": "0s",
        "throttled_millis": 0,
        "requests_per_second": -1.0,
        "throttled_until": "0s",
        "throttled_until_millis": 0,
        "failures": []
    }
}

我的数据是不同的,但配置是相似的.周围75%的数据没有被复制.
我正在使用ElasticSearch的sentence-transformers__msmarco-minilm-l-12-v3
任何帮助?

5hcedyr0

5hcedyr01#

您可能没有足够的处理能力来处理推理处理器,因此,一些文档会出现在failed-collection-with-embeddings索引中,其原因在ingest.failure字段中提到。
您可以使用较小的批处理(在源中指定较小的size)或使用请求节流。

相关问题