任何程序只要可以从标准输入流中读取数据并且可以写入数据到标准输出流就可以通过hadoop流使用其他语言编写mapreduce程序的map函数和reduce函数。map的输出作为reduce的输入。


####使用shell的hadoop流测试：


1 本地新建的input目录中创建3个文件：


ashin@linux:~/test/hadoop/input$ echo "ashin hello blog hadoop" >> f1
ashin@linux:~/test/hadoop/input$ echo "milk test hi blog hadoop" >> f2
ashin@linux:~/test/hadoop/input$ echo "milk miss ashin blog hadoop" >> f3


2 将创建的文件传到hadoop的hdfs上：


ashin@linux:~/test/hadoop/input$ cd ../hadoop-1.1.2/bin/
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -put ../../input/ input1


input1是在hdfs上的目录，默认为user/ashin/input1


ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -ls
Found 5 items
drwxr-xr-x   - ashin supergroup          0 2013-04-22 19:30 /user/ashin/input
drwxr-xr-x   - ashin supergroup          0 2013-04-22 19:35 /user/ashin/input1
drwxr-xr-x   - ashin supergroup          0 2013-04-22 19:21 /user/ashin/output
drwxr-xr-x   - ashin supergroup          0 2013-04-22 19:35 /user/ashin/output1
drwxr-xr-x   - ashin supergroup          0 2013-04-22 19:44 /user/ashin/output2


删除hdfs上的目录：


ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -rmr /user/ashin/input
Deleted hdfs://192.168.2.180:9000/user/ashin/input


3 hadoop流使用shell命令统计文件内容信息：


ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop jar ../contrib/streaming/hadoop-streaming-1.1.2.jar -input input -output output1 -mapper /bin/cat -reducer /usr/bin/wc


4 查看结果(文件行数，单词数，字节数)：


ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -cat output1/*
      3      14      80
cat: File does not exist: /user/ashin/output1/_logs


---------------------------------------------


hadoop流使用shell命令对文件排序：


ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop jar ../contrib/streaming/hadoop-streaming-1.1.2.jar -input input1 -output output3 -mapper /bin/cat -reducer "sort"


查看结果：


ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -cat output3/*
ashin hello blog hadoop
milk miss ashin blog hadoop
milk test hi blog hadoop
cat: File does not exist: /user/ashin/output3/_logs


----------------------------------------------


hadoop流使用shell命令找出文件中含有ashin的文件(貌似含有空格的命令必须用引号引起来才会正常运行)


ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop jar ../contrib/streaming/hadoop-streaming-1.1.2.jar -input input1 -output output2 -mapper /bin/cat -reducer "/bin/grep ashin"


查看结果：


ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -cat output2/*
ashin hello blog hadoop
milk miss ashin blog hadoop
cat: File does not exist: /user/ashin/output2/_logs



------------------------------------------------------------------


####使用Python的hadoop流测试


由于shell命令是linux系统命令，所以无需额外参数，任何可执行文件都可以被指定为mapper/reducer。这些可执行文件不需要事先存放在集群上； 如果在集群上还没有，则需要用-file选项让framework把可执行文件作为作业的一部分。


map.py

#!/usr/bin/env python
#-*- coding:utf-8 -*-
__author__ = 'AshIn'
import sys
for line in sys.stdin:
    words = line.split()
    for word in words:
        print >> sys.stdout, '%s\t%s'%(word, 1)


reduce.py

#!/usr/bin/env python
#-*- coding:utf-8 -*-
__author__ = 'AshIn'
import sys
word_count = {}
for line in sys.stdin:
    word, count = line.split()
    count = word_count.get(word, 0) + int(count)
    word_count[word] = count
for key, value in word_count.items():
    print >> sys.stdout, '%s\t%s'%(key, value)


本地测试python程序是否能够运行(注意先修改脚本权限)：

ashin@linux:~/test/hadoop/input$ cat f1|'./map.py'|'./reduce.py'
ashin1
blog1
hello1
hadoop1


使用hadoop streaming运行，使用-file参数可执行文件可以不传到hdfs上：

ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop jar ../contrib/streaming/hadoop-streaming-1.1.2.jar -input input -output output5 -mapper ../../map.py -reducer ../../reduce.py -file ../../map.py -file ../../reduce.py
ashin@linux:~/test/hadoop/hadoop-1.1.2/bin$ ./hadoop fs -cat output5/*
ashin2
hadoop3
hello1
blog3
hi1
test1
miss1
milk2


这篇文章讲hadoop streaming讲的挺详细：

http://blog.csdn.net/sunlylorn/article/details/8516808


文章为阿小信的个人笔记，转载请注明出处。