如何删除scala中的嵌套列或筛选嵌套列

wvt8vs2t 于 2021-05-27 发布在 Spark

关注(0)|答案(2)|浏览(494)

root
 |-- _id: string (nullable = true)
 |-- h: string (nullable = true)
 |-- inc: string (nullable = true)
 |-- op: string (nullable = true)
 |-- ts: string (nullable = true)
 |-- webhooks: struct (nullable = false)
 |    |    |-- index: string (nullable = false)
 |    |    |-- failed_at: string (nullable = true)
 |    |    |-- status: string (nullable = true)
 |    |    |-- updated_at: string (nullable = true)

如何通过从列表获取输入从（webhooks）中删除列例如filterlist:list[string]=list（“index”，“status”）。有没有什么方法可以像中间模式不会改变最终模式那样迭代行呢

root
 |-- _id: string (nullable = true)
 |-- h: string (nullable = true)
 |-- inc: string (nullable = true)
 |-- op: string (nullable = true)
 |-- ts: string (nullable = true)
 |-- webhooks: struct (nullable = false)
 |    |    |-- index: string (nullable = false)
 |    |    |-- status: string (nullable = true)

scala apache-spark

来源：https://stackoverflow.com/questions/62648200/how-to-drop-nested-column-or-filter-nested-column-in-scala

2条答案

按热度按时间

cclgggtu1#

检查以下代码。

scala> df.printSchema
root
 |-- _id: string (nullable = true)
 |-- h: string (nullable = true)
 |-- inc: string (nullable = true)
 |-- op: string (nullable = true)
 |-- ts: string (nullable = true)
 |-- webhooks: struct (nullable = true)
 |    |-- index: string (nullable = true)
 |    |-- failed_at: string (nullable = true)
 |    |-- status: string (nullable = true)
 |    |-- updated_at: string (nullable = true)

scala> val actualColumns = df.select(s"webhooks.*").columns

scala> val removeColumns = Seq("index","status")

scala> val webhooks = struct(actualColumns.filter(c => !removeColumns.contains(c)).map(c => col(s"webhooks.${c}")):_*).as("webhooks")

输出

scala> df.withColumn("webhooks",webhooks).printSchema
root
 |-- _id: string (nullable = true)
 |-- h: string (nullable = true)
 |-- inc: string (nullable = true)
 |-- op: string (nullable = true)
 |-- ts: string (nullable = true)
 |-- webhooks: struct (nullable = false)
 |    |-- failed_at: string (nullable = true)
 |    |-- updated_at: string (nullable = true)

赞(0）回复(0）举报 2021-05-27

iovurdzv2#

也可以看看https://stackoverflow.com/a/39943812/2204206
在删除嵌套较深的列时会更方便

赞(0）回复(0）举报 2021-05-27