spark流式Dataframe公共到elasticsearch

qltillow  于 2021-07-13  发布在  Spark
关注(0)|答案(1)|浏览(319)

sparksession配置:

val spark = SparkSession.builder()
    .config("es.index.auto.create", "true")
    .config("es.nodes", "1.2.3.4")
    .config("es.port", "9200")
    .config("es.nodes.wan.only", "true")
    .master("local[*]")
    .getOrCreate()

保存查询:

val query = tableDF
    .writeStream
    .format("es")
    .outputMode("append")
    .option("checkpointLocation", "/tmp/checkpoint")
    .option("resource", "spark/speeches")
    .start()

query.awaitTermination()

错误

Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object;

我的依赖项:

compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.0.1'
compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '3.0.1'
compile group: 'org.apache.spark', name: 'spark-streaming_2.12', version: '3.0.1'
implementation group: 'org.elasticsearch', name: 'elasticsearch-hadoop', version: '7.11.1'
6ie5vjzr

6ie5vjzr1#

替换依赖项

implementation group: 'org.elasticsearch', name: 'elasticsearch-hadoop', version: '7.11.1'

implementation group: 'org.elasticsearch', name: 'elasticsearch-spark-20_2.11', version: '7.11.1'

您应该更改其他依赖项以使用scala 2.11

相关问题