org.elasticsearch.hadoop.rest.eshadoopremoteexception:搜索上下文\u缺少\u异常:找不到id的搜索上下文

au9on6nz  于 2021-06-15  发布在  ElasticSearch
关注(0)|答案(1)|浏览(491)

spark任务失败,因为“找不到id的搜索上下文”。我试过几种选择,比如
spark.es.input.max.docs.per.250分区
spark.es.scroll.尺寸100
spark.es.batch.size.bytes 32mb
但任务仍在失败。我们使用的是:spark版本:2.3.1,elasticsearh cluster版本:6.5.4,elasticsearch spark版本:6.5.4

org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: search_context_missing_exception: No search context found for id [4872076]
{"scroll_id":"DXF1ZXJ5QW5kRmV0Y2gBAAAAAABKV4wWRTJHQW9CdEZRTHVqMldEWnQxTUJnQQ=="}
    at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:443)
    at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:400)
    at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:382)
    at org.elasticsearch.hadoop.rest.RestClient.scroll(RestClient.java:458)
    at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:323)
    at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:115)
    at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:61)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:533)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1092)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:347)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:298)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:42)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:336)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:300)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:112)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:384)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

任何帮助都将受到感谢。

i7uaboj4

i7uaboj41#

增加下面参数的值对我有效。
es.scroll.keepalive=120m(默认值为10m)
参考文献:https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html

相关问题