让我们来解决这个问题。

# First we set up the problem

data = [['101,2020-09-04', 30.4, 0.0, 0.0], ['101,2020-09-04', 29.5, 45, 0.0], ['102,2020-09-04', 45.2, 48, 0.0]]
rdd = sc.parallelize(data)

# Break RDD rows into pairs: the common part to serve as our key, and the rest of the list

pair_rdd = rdd.map(lambda x: (x[0], x[1:]))
pair_rdd.collect()

# [('101,2020-09-04', [30.4, 0.0, 0.0]), ('101,2020-09-04', [29.5, 45, 0.0]), ('102,2020-09-04', [45.2, 48, 0.0])]

既然公共部分分开了，我们就可以用它来 reduceByKey ，但首先我们需要一个函数来比较列表值。

# Comparing lists e.g. [1,0] > [0,1] -> True, because it compares elements

# left to right. We reverse the lists in the function, so that the

# zeros are on the left most side

# [0.0, 0.0, 30.4] < [0.0, 45, 29.5] -> True

def get_most_complete(record1, record2):
  if (record1[::-1] > record2[::-1]):
    return record1
  else:
    return record2

# Now we have our function and can reduceByKey

reduced_pair_rdd = pair_rdd.reduceByKey(lambda x,y: get_most_complete(x,y))

# Then we dismantle the pair into its original form

result = reduced_pair_rdd.map(lambda x: [x[0]] + x[1])
result.collect()

# [['101,2020-09-04', 29.5, 45, 0.0], ['102,2020-09-04', 45.2, 48, 0.0]]

这就是你想要的结果。