pythonmapreduce如何添加条件语句

qyswt5oh  于 2021-05-27  发布在  Hadoop
关注(0)|答案(1)|浏览(380)

我是mapreduce的新手,我正在努力寻找movielens 100k数据集中电影的平均影评。我有一个工作程序,可以找到每部电影的平均评论,但我只想对评论超过100的电影这样做。如何添加条件语句来执行此操作?

from mrjob.job import MRJob

class PopularMovieAvgReview(MRJob):
    def mapper(self, key, line):
        (userID, movieID, rating, timestamp) = line.split('\t')
        yield movieID, float(rating)

    def reducer(self, movieID, rating):
        total = 0
        numElements = 0 
        for x in rating:
            total += x
            numElements += 1
        yield movieID, total / numElements

if __name__ == '__main__':
    PopularMovieAvgReview.run()
6pp0gazn

6pp0gazn1#

如果我理解正确,您希望根据给定的评级数量限制输出

def reducer(self, movieID, rating):
    total = 0
    numElements = 0 
    for x in rating:
        total += x
        numElements += 1
    if numElements > 100:
        yield movieID, total / numElements

或者,使用pyspark,您可以聚合,然后过滤评级金额

相关问题