Streaming 提供来Mapreduce的API,允许我们用非JAVA语言编写map和reduce函数,这是我第一次使用Streaming,也是第一次在linux写shell脚本,值得记录一下(2015/12/16)!
我以前习惯用C++写程序,所以这里用C++实现map和reduce。
先介绍下使用Streaming实现的步骤:
1、写map和reduce函数
2、在linux下测试map和reduce函数
3、用hadoop实现
步骤一:
//mapper function
#include<string>
#include<iostream>
using namespace std;
int main()
{
string str;
while(cin>>str)
{
cout<<str<<"\t"<<1<<endl;
}
return 0;
}
//reducer function
#include<iostream>
#include<string>
using namespace std;
int main()
{
string key1,value,key2;
int cnt = 1;
cin>>key1>>value;
while(cin>>key2>>value)
{
if(key1 == key2)
{
cnt++;
}
else
{
cout<<key1<<"\t"<<cnt<<endl;
cnt = 1;
key1 = key2;
}
}
cout<<key2<<"\t"<<cnt<<endl;
return 0;
}
步骤2和3放在wordcount.sh中
#!/bin/bash
#在linux下测试
:<<cc
g++ ./map.cpp -o map
g++ ./reduce.cpp -o reduce
cat /home/hadoop/input/wordcount/wordcount.txt | ./map | sort | ./reduce
cc
#在hadoop下执行
hadoop fs -rmr /user/hadoop/output
hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.2.1.jar \
-input /user/hadoop/input/wordcount/ \
-output /user/hadoop/output/wordcount \
-mapper /home/hadoop/c++/streaming/map \
-reducer /home/hadoop/c++/streaming/reduce \
-file /home/hadoop/c++/streaming/map \
-file /home/hadoop/c++/streaming/reduce