我正尝试像here那样将值从Map转换到 Dataframe 中。
我遇到了一个问题,因为我提取的其中一列有时是double
,有时是integer
。
我是这么做的:
import org.apache.spark.sql.SparkSession
val d: Map[String,Any] = Map(
"count" -> 1,
"results" -> List(
Map("A" -> 20, "C" -> "Hello", "B" -> Map("BA" -> 0, "BC" -> 0)),
Map("A" -> 1.52, "C" -> "Hi", "B" -> Map("BA" -> 0, "BC" -> 0))
),
"parameters" -> Map("P1" -> 805, "P2" -> 20230101)
)
val spark = SparkSession.builder
.master("local")
.appName("Spark app")
.getOrCreate()
import spark.implicits._
val df = d("results")
.asInstanceOf[Seq[Map[String, Any]]]
.map(m =>
(m("A"), m("C")).asInstanceOf[(Double, String)]
)
.toDF("A", "C")
我有:
RuntimeException: Error while encoding: java.lang.ClassCastException: scala.math.BigInt cannot be cast to java.lang.Double
我试过:
val df = d("results")
.asInstanceOf[Seq[Map[String, Any]]]
.map(m =>
(m("A"), m("C")).asInstanceOf[(String, String)]
)
.toDF("A", "C")
但还是:
RuntimeException: Error while encoding: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.String
我怎么才能把所有的东西都转换成String
呢?
1条答案
按热度按时间vbkedwbf1#
Transform Map[String,Any] to a dataframe in Scala
可以用模式匹配
match { case _: Int => ??? ; case _: Double => ??? }
替换强制转换.asInstanceOf[Int]
/.asInstanceOf[Double]
实际上,在引擎盖下,模式匹配是使用
.isInstabceOf
、.asInstanceOf
实现的,因此此模式匹配类似于