无法在PySpark中写入任何内容

l0oc07j2  于 2023-02-15  发布在  Spark
关注(0)|答案(1)|浏览(180)

我尝试将DataFrame写入任何类型的文件格式。我尝试以不同的方式和不同的版本重新安装Spark几次,但每次都收到相同的错误,即使是在另一台机器上。目前在Windows 11本地Hadoop 2.7上使用Spark 3.3.1:

data = [[1, 43, 41], [2, 43, 41], [3, 43, 4]]
x = spark.createDataFrame(data)
x.write.csv('qqq')

收到这个:

File "D:\venvs\spark2\spark_hw.py", line 77, in <module>
    x.write.csv('qqq')
  File "D:\venvs\spark2\lib\site-packages\pyspark\sql\readwriter.py", line 1240, in csv
    self._jwrite.csv(path)
  File "D:\venvs\spark2\lib\site-packages\py4j\java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "D:\venvs\spark2\lib\site-packages\pyspark\sql\utils.py", line 190, in deco
    return f(*a, **kw)
  File "D:\venvs\spark2\lib\site-packages\py4j\protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o44.csv.
: org.apache.spark.SparkException: Job aborted.
mrzz3bfm

mrzz3bfm1#

x.write.format("csv").save("path/where/file/should/go")

将把 Dataframe 写入保存方法中指定路径的csv文件

相关问题