df =spark.createDataFrame([
(78,20,19,90),
],
('Machines',  'Books',  'Vehicles', 'Plants'))

创建一个新的结构列数组，该数组将列名和值名组合在一起。使用magic inline分解结构字段。代码如下

df.withColumn('tab', F.array(*[F.struct(lit(x).alias('Fields'), col(x).alias('Count')).alias(x) for x in df.columns])).selectExpr('inline(tab)').show()

+--------+-----+
|  Fields|Count|
+--------+-----+
|Machines|   78|
|   Books|   20|
|Vehicles|   19|
|  Plants|   90|
+--------+-----+

如unpivot-dataframe tutoral中所述用途：

df = df.selectExpr("""stack(4, "Machines", Machines, "Books", Books, "Vehicles", Vehicles, "Plants", Plants) as (Fields, Count)""")

或者概括一下：

cols = [f'"{c}", {c}' for c in df.columns]
exprs = f"stack({len(cols)}, {', '.join(str(c) for c in cols)}) as (Fields, Count)"
df = df.selectExpr(exprs)

完整示例：

df = spark.createDataFrame(data=[[78,20,19,90]], schema=['Machines','Books','Vehicles','Plants'])

# Hard coded
# df = df.selectExpr("""stack(4, "Machines", Machines, "Books", Books, "Vehicles", Vehicles, "Plants", Plants) as (Fields, Count)""")

# Generalised
cols = [f'"{c}", {c}' for c in df.columns]
exprs = f"stack({len(cols)}, {', '.join(str(c) for c in cols)}) as (Fields, Count)"
df = df.selectExpr(exprs)

[Out]:
+--------+-----+
|Fields  |Count|
+--------+-----+
|Machines|78   |
|Books   |20   |
|Vehicles|19   |
|Plants  |90   |
+--------+-----+