如何在spark2中将struct类型的map转换为json

r9f1avp5 于 2021-05-17 发布在 Spark

关注(0)|答案(1)|浏览(872)

我在数据集中有一个Map字段，其模式如下

|-- party: map (nullable = true)
     |    |-- key: string
     |    |-- value: struct (valueContainsNull = true)
     |    |    |-- partyName: string (nullable = true)
     |    |    |-- cdrId: string (nullable = true)
     |    |    |-- legalEntityId: string (nullable = true)
     |    |    |-- customPartyId: string (nullable = true)
     |    |    |-- partyIdScheme: string (nullable = true)
     |    |    |-- customPartyIdScheme: string (nullable = true)
     |    |    |-- bdrId: string (nullable = true)

需要将其转换为json类型。请建议怎么做。提前谢谢

apache-spark apache-spark-sql apache-spark-2.2

来源：https://stackoverflow.com/questions/64992237/how-to-convert-map-of-struct-type-to-json-in-spark2

1条答案

按热度按时间

siv3szwd1#

spark提供了一个可用于Dataframe操作的json函数：

import org.apache.spark.sql.functions._
import spark.implicits._
val df =
  List(
    ("key1", "party01", "cdrId01"),
    ("key2", "party02", "cdrId02"),
  )
    .toDF("key", "partyName", "cdrId")
    .select(struct($"key", struct($"partyName", $"cdrId")).as("col1"))
    .agg(map_from_entries(collect_set($"col1")).as("map_col"))
    .select($"map_col", to_json($"map_col").as("json_col"))

赞(0）回复(0）举报 2021-05-18

我来回答

如何在spark2中将struct类型的map转换为json

1条答案

相关问题

热门标签

最新问答