使用窗口 row_number 按开划分的函数 ecid 并在 creation_timestamp . Example: ```

sampledata

df=spark.createDataFrame([("ECID-195000300","USER_ID1","2018-08-31 20:00:00"),("ECID-195000300","USER_ID2","2016-08-31 20:00:00")],["ecid","creation_user","creation_timestamp"])

from pyspark.sql import Window
from pyspark.sql.functions import *

w = Window.partitionBy('ecid').orderBy("creation_timestamp")

df.withColumn("rn",row_number().over(w)).filter(col("rn") ==1).drop("rn").show()

+--------------+-------------+-------------------+

| ecid|creation_user| creation_timestamp|

+--------------+-------------+-------------------+

|ECID-195000300| USER_ID2|2016-08-31 20:00:00|

+--------------+-------------+-------------------+

我想你需要一个 window 功能+过滤器。我可以向您提出以下未经测试的解决方案：

import pyspark.sql.window as psw
import pyspark.sql.functions as psf

w = psw.Window.partitionBy('ecid')
df = (df.withColumn("min_tmp", psf.min('creation_timestamp').over(w))
         .filter(psf.col("min_tmp") == psf.col("creation_timestamp")) 
)

这个 window 函数允许您返回 min 在每个 ecid 作为你的新专栏 DataFrame