Streaming（C++）实现WordCount

Streaming 提供来Mapreduce的API,允许我们用非JAVA语言编写map和reduce函数，这是我第一次使用Streaming，也是第一次在linux写shell脚本，值得记录一下（2015/12/16）！

我以前习惯用C++写程序，所以这里用C++实现map和reduce。

先介绍下使用Streaming实现的步骤：

1、写map和reduce函数

2、在linux下测试map和reduce函数

3、用hadoop实现

步骤一：

//mapper function
#include<string>
#include<iostream>

using namespace std;

int main()
{
	string str;
	while(cin>>str)
	{
		cout<<str<<"\t"<<1<<endl;	
	}	
	return 0;
}

//reducer function
#include<iostream>
#include<string>

using namespace std;

int main()
{
	string key1,value,key2;
	int cnt  = 1;
	cin>>key1>>value;
	while(cin>>key2>>value)
	{
		if(key1 == key2)
		{
			cnt++;
		}
		else
		{
			cout<<key1<<"\t"<<cnt<<endl;
			cnt = 1;
			key1 = key2;
		}
	}	
	cout<<key2<<"\t"<<cnt<<endl;
	return 0;
}

步骤2和3放在wordcount.sh中

#!/bin/bash

#在linux下测试
:<<cc
g++ ./map.cpp -o map
g++ ./reduce.cpp -o reduce
cat /home/hadoop/input/wordcount/wordcount.txt | ./map | sort | ./reduce
cc

#在hadoop下执行
hadoop fs -rmr /user/hadoop/output
hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.2.1.jar \
-input /user/hadoop/input/wordcount/ \
-output /user/hadoop/output/wordcount \
-mapper /home/hadoop/c++/streaming/map \
-reducer /home/hadoop/c++/streaming/reduce \
-file /home/hadoop/c++/streaming/map \
-file /home/hadoop/c++/streaming/reduce