你可以简单地使用dropDuplicates()和id作为子集列。
参见下面的示例

# given the following input
# +---+---+---+---+
# | id| c1| c2| c3|
# +---+---+---+---+
# |  1|foo|bar|baz|
# |  1|foo|baz|bar|
# |  2|foo|bar|baz|
# |  2|foo|bar|baz|
# +---+---+---+---+

data_sdf. \
    dropDuplicates(subset=['id']). \
    show()

# +---+---+---+---+
# | id| c1| c2| c3|
# +---+---+---+---+
# |  1|foo|bar|baz|
# |  2|foo|bar|baz|
# +---+---+---+---+

删除重复记录的一种方法是为每个id组向对象框架添加一个行号列，然后只保留每个组的第一行。

验证码：

from pyspark.sql.types import StructType, StructField, IntegerType, StringType
# Define the schema for the dataframe
schema = StructType([
    StructField("id", IntegerType(), True),
    StructField("Name", StringType(), True),
    StructField("Age", IntegerType(), True),
    StructField("Country", StringType(), True)
])

# Create the dataframe
data = [
    (1, "Mike", 18, "USA"),
    (2, "Lily", 25, "Japan"),
    (2, "Lily", 22, "Japan"),
    (3, "John", 23, "China"),
    (4, "Snow", 36, "Korea"),
    (5, "Tom", 28, "Austria"),
    (5, "Cindy", 28, "Italy")
]

df = spark.createDataFrame(data, schema)

# Show the dataframe
df.show()

from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
# Define a window function to partition by id and order by any column
window = Window.partitionBy("id").orderBy("Name")

# Add a row number column to the dataframe
df = df.withColumn("row_number", row_number().over(window))

# Keep only the first row for each id
df = df.filter(df.row_number == 1).drop("row_number")

# Show the resulting dataframe
df.show()

• Window.partitionBy("id").orderBy("Name")：这定义了一个窗口函数，该函数按id列划分子帧，并按Name列对每个分区中的行进行排序。
• df.withColumn("row_number", row_number().over(window))：这将向子框架添加一个名为row_number的新列，其中包含窗口函数定义的每个分区的行号。
• df.filter(df.row_number == 1).drop("row_number")：这将过滤嵌套框，只保留row_number列等于1的行，这对应于每个分区中的第一行。然后，它从生成的嵌套框架中删除row_number列。
  输出：

| ID|名称|年龄|国家|
| --|--|--|--|
| 1 |迈克| 18 |美国|
| 2 |莉莉| 25 |日本|
| 3 |约翰| 23 |中国|
| 4 |雪| 36 |韩国|
| 5 |辛迪| 28 |意大利|