嗨,我想用spark job来吓唬你。当我试图将数据推送到es时,得到下面的错误。我正在用java使用spark结构化的流式kafka
20/07/01 19:00:50 ERROR MicroBatchExecution: Query [id = eb433c87-501c-4c11-9826-e5cf259bda59, runId = 47071902-430c-4656-9918-25b9695c85bc] terminated with error
java.lang.NoSuchMethodError: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object;
at org.elasticsearch.spark.sql.streaming.EsSparkSqlStreamingSink.addBatch(EsSparkSqlStreamingSink.scala:43)
...
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:245)
Exception in thread "stream execution thread for [id = eb433c87-501c-4c11-9826-e5cf259bda59, runId = 47071902-430c-4656-9918-25b9695c85bc]" Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object;
at ...
org.apache.spark.sql.streaming.StreamingQueryException: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object;
下面是代码
results.select(col("value")).writeStream().outputMode(OutputMode.Append())
.format("es")
.option("checkpointLocation", "/tmp/escheckpoint")
.start("index/doc").awaitTermination();
依赖关系详细信息
compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '3.0.0'
compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.0.0'
compile group: 'org.apache.spark', name: 'spark-streaming_2.12', version: '3.0.0'
compile group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.12', version: '3.0.0'
compile group: 'org.apache.spark', name: 'spark-sql-kafka-0-10_2.12', version: '3.0.0'
compile group: 'org.elasticsearch', name: 'elasticsearch-hadoop', version: '7.8.0'
暂无答案!
目前还没有任何答案,快来回答吧!