特定列上的百分位数

uxhixvfz  于 2021-05-29  发布在  Spark
关注(0)|答案(1)|浏览(454)

我有下面的Dataframe。

scala> df.show
+---+------+---+
|  M|Amount| Id|
+---+------+---+
|  1|     5|  1|
|  1|    10|  2|
|  1|    15|  3|
|  1|    20|  4|
|  1|    25|  5|
|  1|    30|  6|
|  2|     2|  1|
|  2|     4|  2|
|  2|     6|  3|
|  2|     8|  4|
|  2|    10|  5|
|  2|    12|  6|
|  3|     1|  1|
|  3|     2|  2|
|  3|     3|  3|
|  3|     4|  4|
|  3|     5|  5|
|  3|     6|  6|
+---+------+---+

创建人

val df=Seq( (1,5,1), (1,10,2), (1,15,3), (1,20,4), (1,25,5), (1,30,6), (2,2,1), (2,4,2), (2,6,3), (2,8,4), (2,10,5), (2,12,6), (3,1,1), (3,2,2), (3,3,3), (3,4,4), (3,5,5), (3,6,6) ).toDF("M","Amount","Id")

这里我有一个基本列m,并根据数量作为id进行排序。我试着把m作为一个整体来计算百分位数,但是每三个数值。
我使用下面的代码来寻找一个群体的百分位数。但是我怎样才能把最后三个值作为目标呢?

df.withColumn("percentile",percentile_approx(col("Amount") ,lit(.5)) over Window.partitionBy("M"))

预期产量

+---+------+---+-----------------------------------+
|  M|Amount| Id| percentile                        |
+---+------+---+-----------------------------------+
|  1|     5|  1| percentile(Amount) whose (Id-1)   |
|  1|    10|  2| percentile(Amount) whose (Id-1,2) |
|  1|    15|  3| percentile(Amount) whose (Id-1,3) |
|  1|    20|  4| percentile(Amount) whose (Id-2,4) |
|  1|    25|  5| percentile(Amount) whose (Id-3,5) |
|  1|    30|  6| percentile(Amount) whose (Id-4,6) |
|  2|     2|  1| percentile(Amount) whose (Id-1)   |
|  2|     4|  2| percentile(Amount) whose (Id-1,2) |
|  2|     6|  3| percentile(Amount) whose (Id-1,3) |
|  2|     8|  4| percentile(Amount) whose (Id-2,4) |
|  2|    10|  5| percentile(Amount) whose (Id-3,5) |
|  2|    12|  6| percentile(Amount) whose (Id-4,6) |
|  3|     1|  1| percentile(Amount) whose (Id-1)   |
|  3|     2|  2| percentile(Amount) whose (Id-1,2) |
|  3|     3|  3| percentile(Amount) whose (Id-1,3) |
|  3|     4|  4| percentile(Amount) whose (Id-2,4) |
|  3|     5|  5| percentile(Amount) whose (Id-3,5) |
|  3|     6|  6| percentile(Amount) whose (Id-4,6) |
+---+------+---+----------------------------------+

这对我来说似乎有点棘手,因为我仍在学习spark.expecting从这里的爱好者的答案。

anauzrmj

anauzrmj1#

添加 orderBy("Amount") 以及 rowsBetween(-2,0) 到窗口定义获取所需结果:
orderby按数量对每个组中的行进行排序
rowsbetween在计算百分位数时仅考虑当前行和前面的两行

val w = Window.partitionBy("M").orderBy("Amount").rowsBetween(-2,0)

df.withColumn("percentile",PercentileApprox.percentile_approx(col("Amount") ,lit(.5))
      .over(w))
  .orderBy("M", "Amount") //not really required, just to make the output more readable
  .show()

印刷品

+---+------+---+----------+
|  M|Amount| Id|percentile|
+---+------+---+----------+
|  1|     5|  1|         5|
|  1|    10|  2|         5|
|  1|    15|  3|        10|
|  1|    20|  4|        15|
|  1|    25|  5|        20|
|  1|    30|  6|        25|
|  2|     2|  1|         2|
|  2|     4|  2|         2|
|  2|     6|  3|         4|
|  2|     8|  4|         6|
|  2|    10|  5|         8|
|  2|    12|  6|        10|
|  3|     1|  1|         1|
|  3|     2|  2|         1|
|  3|     3|  3|         2|
|  3|     4|  4|         3|
|  3|     5|  5|         4|
|  3|     6|  6|         5|
+---+------+---+----------+

相关问题