我在pyspark中读取数据时遇到问题：
数据从aws firehose（示例数据）发送到s3 bucket，存储为json并使用snappy hadoop压缩。
我可以用spark.read.json（）从遗留的spark dataframe中读取数据，但这对使用from\ catalog或from\ options方法的glue dynamic frame（根本不解析架构）不起作用：
spark遗留Dataframe

# import from legacy spark read

spark_df = spark.read.json("s3://my-bucket/sample-json-hadoop-snappy/")
spark_df.printSchema()

- result:
root
 |-- change: double (nullable = true)
 |-- price: double (nullable = true)
 |-- sector: string (nullable = true)
 |-- ticker_symbol: string (nullable = true)
 |-- year: integer (nullable = true)
 |-- dt: date (nullable = true)

粘合动态框架

# import from glue options

options_df = glueContext.create_dynamic_frame.from_options(
  connection_type="s3",
  connection_options = {"paths": ["s3://my-bucket/sample-json-hadoop-snappy/"]},
  format="json"
)
options_df.printSchema()

- result:
root