java中带spark 3.0.0的elasticsearch

jpfvwuh4  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(490)

嗨,我想用spark job来吓唬你。当我试图将数据推送到es时,得到下面的错误。我正在用java使用spark结构化的流式kafka

20/07/01 19:00:50 ERROR MicroBatchExecution: Query [id = eb433c87-501c-4c11-9826-e5cf259bda59, runId = 47071902-430c-4656-9918-25b9695c85bc] terminated with error
java.lang.NoSuchMethodError: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object;
    at org.elasticsearch.spark.sql.streaming.EsSparkSqlStreamingSink.addBatch(EsSparkSqlStreamingSink.scala:43)
    ...
    at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:245)
Exception in thread "stream execution thread for [id = eb433c87-501c-4c11-9826-e5cf259bda59, runId = 47071902-430c-4656-9918-25b9695c85bc]" Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object;
    at ...
org.apache.spark.sql.streaming.StreamingQueryException: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object;

下面是代码

results.select(col("value")).writeStream().outputMode(OutputMode.Append())
                .format("es")
                .option("checkpointLocation", "/tmp/escheckpoint")
                .start("index/doc").awaitTermination();

依赖关系详细信息

compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '3.0.0'
compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.0.0'
compile group: 'org.apache.spark', name: 'spark-streaming_2.12', version: '3.0.0'
compile group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.12', version: '3.0.0'
compile group: 'org.apache.spark', name: 'spark-sql-kafka-0-10_2.12', version: '3.0.0'
compile group: 'org.elasticsearch', name: 'elasticsearch-hadoop', version: '7.8.0'

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题