spark submit-py4j创建sparksession时出错

ogq8wdun  于 2021-05-29  发布在  Spark
关注(0)|答案(0)|浏览(417)

我有一个简单的python文件,它使用

SparkSession.builder.getOrCreate()

看到以下py4j错误:

Traceback (most recent call last):
  File "/home/centos/pysubmit-test.py", line 11, in <module>
    spark = (SparkSession.builder.config(conf=conf).getOrCreate())
  File "/home/centos/pyspark/sql/session.py", line 169, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/home/centos/pyspark/context.py", line 310, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/home/centos/pyspark/context.py", line 115, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/home/centos/pyspark/context.py", line 259, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/home/centos/pyspark/java_gateway.py", line 117, in launch_gateway
    java_import(gateway.jvm, "org.apache.spark.SparkConf")
  File "/home/centos/py4j/java_gateway.py", line 180, in java_import
    return_value = get_return_value(answer, gateway_client, None, None)
  File "/home/centos/py4j/protocol.py", line 332, in get_return_value
    format(target_id, ".", name, value))
py4j.protocol.Py4JError: An error occurred while calling None.None. Trace:
Authentication error: unexpected command.

spark版本:2.1.3
py4j版本:0.10.7
下面是我的bashèu简介。

export PATH=$PATH:$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*:/opt/jdk8u232-b09/bin:/home/centos/maven/bin:/home/centos/spark/bin
export HADOOP_CLASSPATH=$HADOOP_HOME/share/hadoop/tools/lib/*
export JAVA_HOME=/opt/jdk8u232-b09
export SPARK_HOME=/home/centos/spark

知道缺少什么配置吗?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题