scala 什么是好的类型示例来放置一个Map中的浮点值?

qni6mghb  于 2023-02-22  发布在  Scala
关注(0)|答案(1)|浏览(167)

我正尝试像here那样将值从Map转换到 Dataframe 中。
我遇到了一个问题,因为我提取的其中一列有时是double,有时是integer
我是这么做的:

import org.apache.spark.sql.SparkSession

val d: Map[String,Any] = Map(
  "count" -> 1, 
  "results" -> List(
    Map("A" -> 20, "C" -> "Hello", "B" -> Map("BA" -> 0, "BC" -> 0)), 
    Map("A" -> 1.52, "C" -> "Hi", "B" -> Map("BA" -> 0, "BC" -> 0))
  ), 
  "parameters" -> Map("P1" -> 805, "P2" -> 20230101)
)

val spark = SparkSession.builder
  .master("local")
  .appName("Spark app")
  .getOrCreate()

import spark.implicits._

val df = d("results")
  .asInstanceOf[Seq[Map[String, Any]]]
  .map(m => 
    (m("A"), m("C")).asInstanceOf[(Double, String)]
  )
  .toDF("A", "C")

我有:

RuntimeException: Error while encoding: java.lang.ClassCastException: scala.math.BigInt cannot be cast to java.lang.Double

我试过:

val df = d("results")
  .asInstanceOf[Seq[Map[String, Any]]]
  .map(m => 
    (m("A"), m("C")).asInstanceOf[(String, String)]
  )
  .toDF("A", "C")

但还是:

RuntimeException: Error while encoding: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.String

我怎么才能把所有的东西都转换成String呢?

vbkedwbf

vbkedwbf1#

Transform Map[String,Any] to a dataframe in Scala
可以用模式匹配match { case _: Int => ??? ; case _: Double => ??? }替换强制转换.asInstanceOf[Int]/.asInstanceOf[Double]

val df = d("results")
  .asInstanceOf[Seq[Map[String, Any]]]
  .map(m =>
    (
      m("A") match {
        case x: Double => x
        case x: Int    => x.toDouble
      },
      m("C").asInstanceOf[String]
    )
  )
  .toDF("A", "C")

实际上,在引擎盖下,模式匹配是使用.isInstabceOf.asInstanceOf实现的,因此此模式匹配类似于

val x = m("A")
if (x.isInstanceOf[Double])   x.asInstanceOf[Double]
else if (x.isInstanceOf[Int]) x.asInstanceOf[Int].toDouble
else ???

相关问题