如何python安装hadoop_Hadoop 3.0.0-alpha2安装(三）之Python编写MapReduce

4.5.4 Python编写MapReduce

# mkdir -p /data/test && cd /data/test

# vim input.txt

foo foo quux labs foo bar quux abx bar see you bytest welcome test abc labs foo me python hadoop ab ac bc bec python

# vim mapper.py # 刘天斯 著《Python 自动化运维》中的代码

#!/usr/bin/env python

import sys

for line in sys.stdin:

line =line.strip()

words =line.split()

for word inwords:

print'%s\t%s' % (word, 1)

# vim reducer.py # 刘天斯 著《Python 自动化运维》中的代码

#!/usr/bin/env python

from operator import itemgetter

import sys

current_word = None

current_count = 0

word = None

for line in sys.stdin:

line =line.strip()

word,count= line.split('\t', 1)

try:

count =int(count)

except ValueError:

continue

ifcurrent_word == word:

current_count += count

else:

ifcurrent_word:

print '%s\t%s' % (current_word, current_count)

current_count = count

current_word = word

if current_word == word:

print'%s\t%s' % (current_word, current_count)

# chmod +x *.py

# cat input.txt |./mapper.py # 测试py文件

# cat input.txt |./mapper.py |sort -k1,1|./reducer.py

$ hdf dfs -mkdir /user/hduser/input/word

$ hdfs dfs -put /data/test/input.txt/user/hduser/input/word

$ hadoop jar../share/hadoop/tools/lib/hadoop-streaming-3.0.0-alpha2.jar -file/data/test/mapper.py -mapper /data/test/mapper.py -file /data/test/reducer.py-reducer /data/test/reducer.py -input /user/hduser/input/word -output/user/hadoop/word

$ hdfs dfs -ls /user/hadoop/word # 查看生成的文件

Found 2 items

-rw-r--r-- 2hadoop supergroup 0 2017-04-0509:53 /user/hadoop/word/_SUCCESS

-rw-r--r-- 2hadoop supergroup 117 2017-04-0509:53 /user/hadoop/word/part-00000

$ hdfs dfs -cat/user/hadoop/word/part-00000

ab 1

abc 1

abx 1

ac 1

bar 2

bc 1

bec 1

by 1

foo 4

hadoop 1

labs 2

me 1

python 2

quux 2

see 1

test 2

welcomne 1

you 1

### good 成功了，继续努力……

### 以上python代码也可以这里看到：http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/