我有一个Python中的Glue任务，它将单个的parquet文件写入S3中的Delta Lake。

import sys

from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext

raw_parquet_path = "s3://my-bucket/data/raw-parquet/motor/"
delta_table_path = (
    "s3://my-bucket/data/delta-tables/my_table/"
)
partition_list = ["_event_id"]

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Script generated for node S3 bucket
S3bucket_node1 = glueContext.create_dynamic_frame.from_options(
    format_options={},
    connection_type="s3",
    format="parquet",
    connection_options={
        "paths": [raw_parquet_path],
        "recurse": True,
    },
    transformation_ctx="S3bucket_node1",
)

# Script generated for node sink_to_delta_lake
additional_options = {
    "path": delta_table_path,
    "write.parquet.compression-codec": "snappy",
    "mergeSchema": "true",
}
sink_to_delta_lake_node3_df = S3bucket_node1.toDF()
sink_to_delta_lake_node3_df.write.format("delta").options(
    **additional_options
).partitionBy(*partition_list).mode("append").save()

job.commit()

我出错了

An error occurred while calling o111.save. Failed to find data source: delta. Please find packages at

我添加了作业详细信息->高级属性->作业参数

• --conf：spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
• --packages：io.delta:delta-core_2.12:2.1.0

(Note --conf基于this在一行）

然而，错误变成了
调用o100.getDynamicFrame时出错。无法找到目录“spark_catalog”的目录插件类：org.apache.spark.sql.delta.catalog.DeltaCatalog
有什么想法吗？谢谢！