python 执行 hadoop-2.2.0 mapreduce

简单记录一下

需求大概是把一类文本格式转换成另外一类文本格式，转换前是 如下这样：

1.245.152.25	[2,4,12,17,888,15334,23,6]
1.245.152.250	[2,4,11,37,814,33374,23,6]
1.245.152.251	[6,14,16,65,918,40281,1433,17]
1.245.152.252	[0,2,6,0,420,0,23,2]
1.245.152.253	[0,6,10,4,698,304,23,6]
1.245.152.254	[0,8,8,8,512,432,23,7]
1.245.152.255	[2,0,0,12,0,10824,15626,2]
1.245.152.26	[4,9,14,38,983,6970,22,13]
1.245.152.27	[13,0,38,18,4147,5459,35342,9]

希望修改后是这样的：

1.245.152.25 2 4 12 17 888 15334 23 6	
1.245.152.250 2 4 11 37 814 33374 23 6	
1.245.152.251 6 14 16 65 918 40281 1433 17	
1.245.152.252 0 2 6 0 420 0 23 2	
1.245.152.253 0 6 10 4 698 304 23 6	
1.245.152.254 0 8 8 8 512 432 23 7	
1.245.152.255 2 0 0 12 0 10824 15626 2	
1.245.152.26 4 9 14 38 983 6970 22 13	
1.245.152.27 13 0 38 18 4147 5459 35342 9	

mapper.py

#!/usr/bin/env python

import sys

for line in sys.stdin:
    line = line.strip()
    try:
        (ip,datalist)=line.split('\t',1)
        datalist = datalist[1:-1]
        datas = datalist.split(',')
        ll=ip
        for i in datas:
            ll = ll+' '+i
        print ll
    except Exception:
#	print ip,datalist
        continue

执行命令：

./hadoop jar ./hadoop-streaming-2.2.0.jar -mapper "python ./tmp.py" -file ./tmp.py  -input /input/* -output /output7

(后来又不可以了，改成了如下的样子)
./hadoop jar ./hadoop-streaming*.jar -D stream.non.zero.exit.is.failure=false -mapper "python /home/mymiss/xxxx/hadoop-2.2.0/bin/tmp.py" -file ./tmp.py -input /test_input -output /test_output

执行结果：

最开始执行命令为如下时，一直报错：

./hadoop jar ./hadoop-streaming-2.2.0.jar -file ./tmp.py -mapper ./tmp.py  -input /input/* -output /output7

错误如下：

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

因为要指名是 python 来执行这些 streaming 吧...

from MXY I learnt to read help docs: wiki.apache.org/hadoop/HadoopStreaming