我试图连接与水在r的Spark和Spark版本安装和它与水的兼容性问题。我正在一个emr集群上以“yarn client”模式运行。下面的代码显示了我得到的错误。

install.packages("sparklyr")
install.packages("RCurl")

library(sparklyr)
spark_install(version = "3.0.1")
Sys.setenv("SPARK_HOME" = "/usr/lib/spark/")

install.packages("h2o", type = "source", repos = "https://h2o-release.s3.amazonaws.com/h2o/rel-zermelo/2/R")

# install.packages("h2o", type="source", repos="https://h2o-release.s3.amazonaws.com/h2o/rel-zahradnik/6/R")

install.packages("rsparkling", type = "source", repos = "http://h2o-release.s3.amazonaws.com/sparkling-water/spark-3.0/3.32.0.2-1-3.0/R")

options(rsparkling.sparklingwater.version = "3.0.1")
library(rsparkling)
config <- spark_config()

# Set dynamicAllocation.enabled to false for H2O.

config$spark.dynamicAllocation.enabled <- "false"

sc <- spark_connect(master = "yarn-client", version = "3.0.1", config=config )
spark_version(sc)

输出状态：

> sc <- spark_connect(master = "yarn-client", version = "3.0.1", config=config )
> spark_version(sc)
[1] ‘2.4.4’

它会在h2o中产生以下错误：

> hc <- H2OContext.getOrCreate()
Error: ai.h2o.sparkling.backend.exceptions.WrongSparkVersion: You are trying to use Sparkling Water built for Spark 3.0, but your Spark is of version 2.4.4. Please make sure to use correct Sparkling Water for your Spark and re-run the application.