我有一个脚本，试图加载一个pyspark Dataframe 到aws s3。我已经验证了s3中启用了公共访问，一位同事设法从data bricks中的脚本将文件上传到bucket。然而，当我尝试将 Dataframe 从本地机器上运行的进程保存到存储桶时，我得到以下错误：

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden;

我使用的是pyspark版本3.2.1和python 3.10。以下是我的配置：

from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
from pyspark.sql.types import StructField, StructType, IntegerType, StringType
import os
import sys

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable

conf = SparkConf().set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.2.0')
conf.set('spark.hadoop.fs.s3a.aws.credentials.provider', 'org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider')

sc = SparkContext(conf=conf)
sc.setSystemProperty('com.amazonaws.services.s3.enableV4', 'true')

# Setting up S3 bucket credentials
ACCESS_KEY_ID = "xxxxxxx"
SECRET_ACCESS_KEY = "xxxxxxx"
AWS_BUCKET_NAME = "bucket_name"
DIRECTORY = "/directory /"

# hadoopConf = sc._jsc.hadoopConfiguration()
sc._jsc.hadoopConfiguration().set('fs.s3a.access.key', ACCESS_KEY_ID)
sc._jsc.hadoopConfiguration().set('fs.s3a.secret.key', ENCODED_SECRET_KEY)
sc._jsc.hadoopConfiguration().set('fs.s3a.endpoint', 's3-eu-central-1.amazonaws.com')
sc._jsc.hadoopConfiguration().set('fs.s3a.impl', 'org.apache.hadoop.fs.s3a.S3AFileSystem')

spark = SparkSession(sc)

下面是尝试保存 Dataframe 的代码行：

result_dataframe.write.format("csv").option("header", "true").option("delimiter", "\t").mode("append").save(f"s3a://bucket_name/directory/{filename}.csv")

我不明白为什么我的访问被拒绝了。任何帮助将不胜感激。