如何改变Spark壳中的Spark环节?

yftpprvb  于 2023-01-21  发布在  Apache
关注(0)|答案(1)|浏览(103)

我一直在尝试使用同一个spark上下文进行多个spark会话,一旦spark-shell启动,我就可以使用以下代码创建一个新的spark会话。

val spark2 = spark.newSession();
spark2: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea

scala> spark
res0: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@1390db9e

scala> spark2
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea

然而,即使会话已经创建并可用,它仍然不是一个活动会话,每个Spark会话都有自己的临时视图,如果我正在创建一个 Dataframe ,它已经与默认的Spark会话相关联。

scala> val s = Seq(1,2,3).toDF("num")
s: org.apache.spark.sql.DataFrame = [num: int]

scala> val s2 = Seq(4,5,6).toDF("num")
s2: org.apache.spark.sql.DataFrame = [num: int]

scala> s
res2: org.apache.spark.sql.DataFrame = [num: int]

scala> s.sparkSession
res3: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@1390db9e

是否有办法切换活动的spark会话,以便使用该会话完成下一步操作?

6pp0gazn

6pp0gazn1#

您可能正在寻找在companion object中定义的setActiveSession(session: SparkSession): Unit方法。

scala> val spark2 = spark.newSession();
spark2: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea

scala> import org.apache.spark.sql.SparkSession

scala> SparkSession.setActiveSession(spark2)

scala> SparkSession.active
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea

相关问题