今天,我们利用python编写一个MapReduce程序,程序的目的还是百年不变的计算单词个数,也就是WordCunt。
所谓mapreduce其实就是先分散计算后综合处理计算结果。
首先我们来看一下map部分的代码。
#!/usr/bin/env python
import sys
# input comes from STDIN (standard input)
for line in sys.stdin:
# remove leading and trailing whitespace
line = line.strip()
# split the line into words
words = line.split()
# increase counters
for word in words:
# write the results to STDOUT (standard output);
# what we output here will be the input for the
# Reduce step, i.e. the input for reducer.py
#
# tab-delimited; the trivial word count is 1
print '%s\t%s' % (word, 1)
mapper其实只做了一个功能,就是读取每行文字,然后分割成一个一个的单词 <