我在ElasticSearch中尝试语义搜索,遵循这个tutorial。
当我复制一个索引文件到另一个索引[重新索引]以下命令
POST _reindex?wait_for_completion=false
{
"source": {
"index": "collection"
},
"dest": {
"index": "collection-with-embeddings",
"pipeline": "text-embeddings"
}
}
有些文件在新索引中丢失了。但我不知道原因。我正在努力找出原因。
关于上下文,
PUT _ingest/pipeline/text-embeddings
{
"description": "Text embedding pipeline",
"processors": [
{
"inference": {
"model_id": "sentence-transformers__msmarco-minilm-l-12-v3",
"target_field": "text_embedding",
"field_map": {
"text": "text_field"
}
}
}
],
"on_failure": [
{
"set": {
"description": "Index document to 'failed-<index>'",
"field": "_index",
"value": "failed-{{{_index}}}"
}
},
{
"set": {
"description": "Set error message",
"field": "ingest.failure",
"value": "{{_ingest.on_failure_message}}"
}
}
]
}
这是任务详细信息
{
"completed": true,
"task": {
"node": "YgR8udaSSMqClwCGWOBGBw",
"id": 5946104,
"type": "transport",
"action": "indices:data/write/reindex",
"status": {
"total": 2414,
"updated": 1346,
"created": 1068,
"deleted": 0,
"batches": 3,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1.0,
"throttled_until_millis": 0
},
"description": "reindex from [source_index] to [destination_index]",
"start_time_in_millis": 1680795982705,
"running_time_in_nanos": 22702121635,
"cancellable": true,
"cancelled": false,
"headers": {}
},
"response": {
"took": 22699,
"timed_out": false,
"total": 2414,
"updated": 1346,
"created": 1068,
"deleted": 0,
"batches": 3,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled": "0s",
"throttled_millis": 0,
"requests_per_second": -1.0,
"throttled_until": "0s",
"throttled_until_millis": 0,
"failures": []
}
}
我的数据是不同的,但配置是相似的.周围75%的数据没有被复制.
我正在使用ElasticSearch的sentence-transformers__msmarco-minilm-l-12-v3
。
任何帮助?
1条答案
按热度按时间5hcedyr01#
您可能没有足够的处理能力来处理推理处理器,因此,一些文档会出现在
failed-collection-with-embeddings
索引中,其原因在ingest.failure
字段中提到。您可以使用较小的批处理(在源中指定较小的
size
)或使用请求节流。