有一个使用arrays_zip的简单得多的解决方案，不需要explode/collect_list（对于复杂数据，这可能容易出错/很困难，因为它依赖于使用类似id列的东西）：

>>> from pyspark.sql import Row
>>> from pyspark.sql.functions import arrays_zip
>>> df = sc.createDataFrame((([Row(x=1, y=2, z=3), Row(x=2, y=3, z=4)],),), ['array_of_structs'])
>>> df.show(truncate=False)
+----------------------+
|array_of_structs      |
+----------------------+
|[{1, 2, 3}, {2, 3, 4}]|
+----------------------+
>>> df.printSchema()
root
 |-- array_of_structs: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- x: long (nullable = true)
 |    |    |-- y: long (nullable = true)
 |    |    |-- z: long (nullable = true)
>>> # Selecting only two of the nested fields:
>>> selected_df = df.select(arrays_zip("array_of_structs.x", "array_of_structs.y").alias("array_of_structs"))
>>> selected_df.printSchema()
root
 |-- array_of_structs: array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- x: long (nullable = true)
 |    |    |-- y: long (nullable = true)
>>> selected_df.show()
+----------------+
|array_of_structs|
+----------------+
|[{1, 2}, {2, 3}]|
+----------------+

EDIT添加相应的Spark SQL代码，因为这是OP要求的：

>>> df.createTempView("test_table")
>>> sql_df = sc.sql("""
SELECT
transform(array_of_structs, x -> struct(x.x, x.y)) as array_of_structs
FROM test_table
""")
>>> sql_df.printSchema()
root
 |-- array_of_structs: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- x: long (nullable = true)
 |    |    |-- y: long (nullable = true)
>>> sql_df.show()
+----------------+
|array_of_structs|
+----------------+
|[{1, 2}, {2, 3}]|
+----------------+

事实上，我提供的伪代码是有效的。对于对象的嵌套数组来说，这不是那么简单。首先，数组应该被分解（EXPLODE（）函数），然后选择一个子集。之后，就可以创建一个COLLECT_LIST（）。

WITH
  unfold_by_items AS (SELECT id, EXPLODE(Items) AS item FROM spark_tbl_items)
, format_items as (SELECT
    id
    , STRUCT(
              item.item_id
            , item.name
        ) AS item
    FROM unfold_by_items)
, fold_by_items AS (SELECT id, COLLECT_LIST(item) AS Items FROM format_items GROUP BY id)

SELECT * FROM fold_by_items

这将只从Items中的结构中选择两个字段，并最终返回一个数据集，该数据集再次包含一个具有Items的数组。