mrjob:-在map reduce中显示中间值

9q78igpj  于 2021-06-04  发布在  Hadoop
关注(0)|答案(1)|浏览(441)

在使用python mrjob库运行mapreduce程序时,如何在终端上显示中间值(即打印变量或列表)?

xriantvc

xriantvc1#

可以使用sys.stderr.write()将结果输出到标准错误。举个例子:

from mrjob.job import MRJob
import sys
class MRWordCounter(MRJob):
    def mapper(self, key, line):
        sys.stderr.write("MAPPER INPUT: ({0},{1})\n".format(key,line))
        for word in line.split():
            yield word, 1

    def reducer(self, word, occurrences):
        occurencesList= list(occurrences)
        sys.stderr.write("REDUCER INPUT: ({0},{1})\n".format(word,occurencesList))
        yield word, sum(occurencesList)

if __name__ == '__main__':
    MRWordCounter.run()

相关问题