elasticsearch Logstash opensearch输出没有获取document_id

czfnxgou  于 2023-03-22  发布在  ElasticSearch
关注(0)|答案(2)|浏览(146)

我正在尝试使用以下命令在两个opensearch示例之间传输数据:

input {
    opensearch {
        hosts =>  ["http://192.168.1.245:8443"]
        index =>  "crawl"
        query =>  '{ "query": { "match_all": {}} }'
    }
}

output {
  opensearch {
    hosts => ["http://192.168.1.245:9200"]
    index => "crawl"
    document_id => "%{[id]}"
    doc_as_upsert => true 
  }
  stdout { codec => rubydebug { metadata => true } }
}

我遇到的问题是它没有将输入document_id插入到输出document_id中:
我试过:

document_id => "%{[id]}"
document_id => "%{id}"
document_id => "%{[_id]}"
document_id => "%{_id}"
document_id => "%{[document_id]}"
document_id => "%{document_id}"
document_id => "%{[metadata][id]}"

它将其保存到索引中作为查找:

"_index": "crawl",
"_id": "%{document_id}",
"_score": 1.0,

有人知道怎么让它工作吗?

jogvjijk

jogvjijk1#

你需要这样做:

  • 在输入中添加docinfo => true
  • [@metadata][_id]字段中查找文档的_id

配置:

input {
    opensearch {
        hosts =>  ["http://192.168.1.245:8443"]
        index =>  "crawl"
        query =>  '{ "query": { "match_all": {}} }'
        docinfo => true                           <----------
    }
}

output {
  opensearch {
    hosts => ["http://192.168.1.245:9200"]
    index => "crawl"
    document_id => "%{[@metadata][input][opensearch][_id]}"          <----------
    doc_as_upsert => true 
  }
  stdout { codec => rubydebug { metadata => true } }
}
9lowa7mx

9lowa7mx2#

根据Val给出的建议,我设法让它与以下内容一起工作:

input {
    opensearch {
        hosts =>  ["http://192.168.1.245:8443"]
        index =>  "crawl"
        query =>  '{ "query": { "match_all": {}} }'
        docinfo => true <----
        docinfo_target => "[@metadata][doc]" <----
    }
}

output {
  opensearch {
    hosts => ["http://192.168.1.245:9200"]
    index => "crawl"
    document_id => "%{[@metadata][doc][_id]}" <----
    doc_as_upsert => true 
  }
  stdout { codec => rubydebug { metadata => true } }
}

相关问题