在spark 3.x中，以下属性 spark.sql.legacy.setCommandRejectsSparkCoreConfs 新引入的默认情况下拒绝spark core遗留配置更改(https://github.com/apache/spark/blob/d897825d2d0430cb52ae9ac0f6fd742582041682/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/sqlconf.scala) [line:2353]这在spark 2.x版本中不可用。
这是属性的代码片段，

val SET_COMMAND_REJECTS_SPARK_CORE_CONFS =
    buildConf("spark.sql.legacy.setCommandRejectsSparkCoreConfs")
      .internal()
      .doc("If it is set to true, SET command will fail when the key is registered as " +
        "a SparkConf entry.")
      .version("3.0.0")
      .booleanConf
      .createWithDefault(true)

默认情况下，启动spark 3.x时，

scala> spark.conf.get("spark.sql.legacy.setCommandRejectsSparkCoreConfs")
res0: String = true

这就导致了你的失败

scala> spark.conf.set("spark.driver.host","192.168.93.131")
org.apache.spark.sql.AnalysisException: Cannot modify the value of a Spark config: spark.driver.host;
  at org.apache.spark.sql.RuntimeConfig.requireNonStaticConf(RuntimeConfig.scala:158)
  at org.apache.spark.sql.RuntimeConfig.set(RuntimeConfig.scala:42)
  ... 47 elided

解决方案：尝试设置 spark.sql.legacy.setCommandRejectsSparkCoreConfs,false 然后继续更改 spark.driver.host 你可以改变它。

scala> spark.conf.set("spark.sql.legacy.setCommandRejectsSparkCoreConfs","false")

scala> spark.conf.getAll("spark.driver.host")
res3: String = 192.168.93.131

scala> spark.conf.set("spark.driver.host","192.168.93.132")

scala> spark.conf.set("spark.driver.host","192.168.93.132")
scala> spark.conf.getAll("spark.driver.host")
res6: String = 192.168.93.132