scala—遍历< iterable>并在自定义数据类型中填充向量

33qvvth1  于 2021-05-24  发布在  Spark
关注(0)|答案(1)|浏览(586)

这是我的第一篇文章,我需要一点帮助,在scala编程任务中,这不是那么简单(至少对我来说)。
我在2.10版中使用scala,在spark3.0.0-preview2版本下。
从mysql数据库导入的数据类型如下:

95,118.37,118.47,111.725,114.3,1049181,AMP,2020-04-14
96,116.88,117.84,113.11,114.92,827085,AMP,2020-04-13
97,113.64,124.61,113.64,120.47,1608575,AMP,2020-04-09
98,104.48,112.48,102.28,111.69,996230,AMP,2020-04-08
99,109.17,112.23,102.41,103.48,1302910,AMP,2020-04-07
100,42.25,42.25,41.73,41.82,639964,G,2020-08-26
101,41.98,42.15,41.76,42.12,501219,G,2020-08-25
102,41.52,42.015,41.45,41.9,479076,G,2020-08-24
103,41.27,41.46,40.99,41.16,752730,G,2020-08-21
104,41.74,41.965,41.25,41.3,596435,G,2020-08-20
105,42.14,42.21,41.87,41.94,422493,G,2020-08-19

然后,通过Map过程,这些数据将被重新格式化为以下类型的tuple2<key,value>:

(AMP,(1,156.77,156.915,155.03,155.74,527938,AMP,2020-08-26))
(AMP,(2,159.48,159.88,156.86,156.99,535905,AMP,2020-08-25))
(AMP,(3,155.38,157.75,155.33,157.72,758272,AMP,2020-08-24))
(AMP,(4,155.24,156.79,153.92,154.51,653496,AMP,2020-08-21))
(AMP,(5,155.24,157.39,154.27,155.14,516138,AMP,2020-08-20))
(AMP,(6,156.65,160.06,156.57,156.85,577637,AMP,2020-08-19))
(AMP,(7,158.05,158.35,156.34,156.5,544429,AMP,2020-08-18))
(AMP,(8,159.69,159.82,157.76,157.83,437624,AMP,2020-08-17))

其中,每个记录的类型为:

org.apache.spark.rdd.RDD[(String, (Int, Double, Double, Double, Double, Int, String, String))]

然后,我需要对所有键进行分组,并编写了一个“groupbykey”过程:

val SA = Simboli.groupByKey

导致变量:

org.apache.spark.rdd.RDD[(String, Iterable[(Int, Double, Double, Double, Double, Int, String, String)])]

我现在的问题是:我可以创建一个类型为“vector”或“sequence”的新变量,在vector列表中插入这种奇怪类型的每一条记录吗?
例如,每一项都是新的向量:

RDD[(String, Iterable[(Int,   .....

我发现的唯一方法就是用这种方式转换这种变量:
只取第一个“组”

val SAG : Array[(String, Iterable[(Int, Double, 
                                   Double, Double, 
                                   Double, Int, 
                                   String, String)])] = SA.take(1);

提取“iterable”部分:

val SAGITB : Array[Iterable[(Int, Double, 
                             Double, Double, 
                             Double, Int, 
                             String, String)]] = SAG.map(item => item._2);

转换“iterator”中的“iterable”:

val SAGITT : Array[Iterator[(Int, Double, 
                             Double, Double, 
                             Double, Int, 
                             String, String)]] = SAGITB.map(item => item.iterator);

提取值:

val SARDD : Array[(Int, Double, 
                   Double, Double, 
                   Double, Int, 
                   String, String)] = SAGITT.map(item => item.next);

最后,我尝试用每一项填充一个向量或序列,插入一个for循环,但我做不到。这是我最后一次尝试:

val SV3 : Vector[Array[(Int, Double, Double, 
                        Double, Double, Int, 
                        String, String)]] = Vector.empty; 

for (it <- 0 to 20){

  println("Riga numero: " + it);

  SV3 :+ SAGITT.map(item => item.next);

}

最后,我的问题是:如何用“iterable”或“iterator”类型的数据填充向量或序列,或者,如何从iterable的rdd中提取所有数据,用这些数据转换并填充一个简单的向量?
非常感谢!!

wwwo4jvm

wwwo4jvm1#

这相当简单。
首先需要转换嵌套的 Iterable 回到一个 Array :

val SV3A : Array[Array[(Int, Double, Double,
  Double, Double, Int,
  String, String)]] = SAGITB.map(_.toArray)

然后你可以转换成 Vector 像这样,使用 :_* 扩展 Array 到构造函数的单独参数中:

val SV3 : Vector[Array[(Int, Double, Double,
  Double, Double, Int,
  String, String)]] = Vector(SV3A:_*)

或者你可以这样一步将两者结合起来:

val SV3 = Vector(SAGITB.map(_.toArray):_*)

或者更简单的说

val SV3 = SAGITB.map(_.toArray).toVector

相关问题