如何计算PySpark Dataframe 中每个键的百分位数？

wfypjpf4 于 2023-06-21 发布在 Spark

关注(0)|答案(3)|浏览(135)

我有一个PySpark Dataframe ，由三列x，y，z组成。
X在此 Dataframe 中可能有多行。我如何分别计算x中每个键的百分位数？

+------+---------+------+
|  Name|     Role|Salary|
+------+---------+------+
|   bob|Developer|125000|
|  mark|Developer|108000|
|  carl|   Tester| 70000|
|  carl|Developer|185000|
|  carl|   Tester| 65000|
| roman|   Tester| 82000|
| simon|Developer| 98000|
|  eric|Developer|144000|
|carlos|   Tester| 75000|
| henry|Developer|110000|
+------+---------+------+

所需输出：

+------+---------+------+---------+
|  Name|     Role|Salary|      50%|
+------+---------+------+---------+
|   bob|Developer|125000|117500.0 |
|  mark|Developer|108000|117500.0 |
|  carl|   Tester| 70000|72500.0  |
|  carl|Developer|185000|117500.0 |
|  carl|   Tester| 65000|72500.0  |
| roman|   Tester| 82000|72500.0  |
| simon|Developer| 98000|117500.0 |
|  eric|Developer|144000|117500.0 |
|carlos|   Tester| 75000|72500.0  |
| henry|Developer|110000|117500.0 |
+------+---------+------+---------+

pyspark

来源：https://stackoverflow.com/questions/57424211/how-compute-the-percentile-in-pyspark-dataframe-for-each-key

3条答案

按热度按时间

jobtbby31#

尝试groupby + F.expr：

import pyspark.sql.functions as F

df1 = df.groupby('Role').agg(F.expr('percentile(Salary, array(0.25))')[0].alias('%25'),
                             F.expr('percentile(Salary, array(0.50))')[0].alias('%50'),
                             F.expr('percentile(Salary, array(0.75))')[0].alias('%75'))
df1.show()

输出：

+---------+--------+--------+--------+
|     Role|     %25|     %50|     %75|
+---------+--------+--------+--------+
|   Tester| 68750.0| 72500.0| 76750.0|
|Developer|108500.0|117500.0|139250.0|
+---------+--------+--------+--------+

现在，您可以将df1与原始 Dataframe 连接起来：

df.join(df1, on='Role', how='left').show()

输出：

+---------+------+------+--------+--------+--------+
|     Role|  Name|Salary|     %25|     %50|     %75|
+---------+------+------+--------+--------+--------+
|   Tester|  carl| 70000| 68750.0| 72500.0| 76750.0|
|   Tester|  carl| 65000| 68750.0| 72500.0| 76750.0|
|   Tester| roman| 82000| 68750.0| 72500.0| 76750.0|
|   Tester|carlos| 75000| 68750.0| 72500.0| 76750.0|
|Developer|   bob|125000|108500.0|117500.0|139250.0|
|Developer|  mark|108000|108500.0|117500.0|139250.0|
|Developer|  carl|185000|108500.0|117500.0|139250.0|
|Developer| simon| 98000|108500.0|117500.0|139250.0|
|Developer|  eric|144000|108500.0|117500.0|139250.0|
|Developer| henry|110000|108500.0|117500.0|139250.0|
+---------+------+------+--------+--------+--------+

赞(0）回复(0）举报 2023-06-21

enyaitl32#

array实际上不是必需的：

F.expr('percentile(Salary, 0.5)')

与窗口函数一起，它完成了以下工作：

df = df.withColumn('50%', F.expr('percentile(Salary, 0.5)').over(W.partitionBy('Role')))

df.show()
#  +------+---------+------+--------+
#  |  Name|     Role|Salary|     50%|
#  +------+---------+------+--------+
#  |   bob|Developer|125000|117500.0|
#  |  mark|Developer|108000|117500.0|
#  |  carl|Developer|185000|117500.0|
#  | simon|Developer| 98000|117500.0|
#  |  eric|Developer|144000|117500.0|
#  | henry|Developer|110000|117500.0|
#  |  carl|   Tester| 70000| 72500.0|
#  |  carl|   Tester| 65000| 72500.0|
#  | roman|   Tester| 82000| 72500.0|
#  |carlos|   Tester| 75000| 72500.0|
#  +------+---------+------+--------+

赞(0）回复(0）举报 2023-06-21

iqih9akk3#

您可以尝试spark中提供的approxQuantile功能。
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.approxQuantile.html

赞(0）回复(0）举报 2023-06-21