hadoop C++ pipes 运行实现

最新推荐文章于 2021-12-03 17:04:49 发布

oizz01

最新推荐文章于 2021-12-03 17:04:49 发布

阅读量1k

点赞数

分类专栏： hadoop 文章标签： hadoop C++ pipes

本文链接：https://blog.csdn.net/oizz01/article/details/16903039

版权

hadoop 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

花了好长时间，调掉所有的bug，终于把hadoop pipes C++实现弄好了。写篇文章来祭奠下逝去的光阴，阿门。

准备：

1.按照教程：http://blog.sina.com.cn/s/blog_9c43254d0101ngug.html 安装好hadoop并可以jps到6个

2.terminal熟悉

开始：

1.先写好C++程序，这里忘记从哪copy来的一份了，反正可以用就好了,我的名字叫main.cpp

#include <algorithm>
#include <limits>
#include <string>
 
#include  "stdint.h"  // <--- to prevent uint64_t errors!
 
#include "hadoop/Pipes.hh"
#include "hadoop/TemplateFactory.hh"
#include "hadoop/StringUtils.hh"
 
using namespace std;

class WordCountMapper : public HadoopPipes::Mapper {
public:
  // constructor: does nothing
  WordCountMapper( HadoopPipes::TaskContext& context ) {
  }

  // map function: receives a line, outputs (word,"1")
  // to reducer.
  void map( HadoopPipes::MapContext& context ) {
    //--- get line of text ---
    string line = context.getInputValue();

    //--- split it into words ---
    vector< string > words =
      HadoopUtils::splitString( line, " " );

    //--- emit each word tuple (word, "1" ) ---
    for ( unsigned int i=0; i < words.size(); i++ ) {
      context.emit( words[i], HadoopUtils::toString( 1 ) );
    }
  }
};
 
class WordCountReducer : public HadoopPipes::Reducer {
public:
  // constructor: does nothing
  WordCountReducer(HadoopPipes::TaskContext& context) {
  }

  // reduce function
  void reduce( HadoopPipes::ReduceContext& context ) {
    int count = 0;

    //--- get all tuples with the same key, and count their numbers ---
    while ( context.nextValue() ) {
      count += HadoopUtils::toInt( context.getInputValue() );
    }

    //--- emit (word, count) ---
    context.emit(context.getInputKey(), HadoopUtils::toString( count ));
  }
};
 
int main(int argc, char *argv[]) {
  return HadoopPipes::runTask(HadoopPipes::TemplateFactory<
                              WordCountMapper,
                              WordCountReducer >() );
}

2.在和main.cpp同目录底下新建makefile文件，内容如下：

CC = g++
HADOOP_INSTALL = /usr/local/hadoop
PLATFORM = Linux-i386-32
CPPFLAGS = -m32 -I$(HADOOP_INSTALL)/c++/$(PLATFORM)/include -I$(HADOOP_INSTALL)/src/c++/install/include -L$(HADOOP_INSTALL)/src/c++/install/lib -lhadooputils -lhadooppipes -lcrypto -lssl -lpthread

wordcount: main.cpp
	$(CC) $(CPPFLAGS) $< -Wall -L$(HADOOP_INSTALL)/c++/$(PLATFORM)/lib -lhadooppipes \
	-lhadooputils -lcrypto -lpthread -g -O2 -o $@

注意我的hadoop_install目录可能和你的不一样，还有platform也可能不一样，自己去搜怎么确定这两个的值

然后make产生可执行文件wordcount

hello.txt

hello world

3.配置hadoop远端文件：

A.在/etc/environment中加入PATH路径 /usr/local/hadoop/bin ，这个是我的安装目录底下bin文件夹，为的是方便后面操作

B. 运行./start-all.sh开始hadoop远端操作

C. 运行如下脚本

hadoop dfs -mkdir /home

hadoop dfs -mkdir /bin

意思是在hadoop远端根目录地下产生两个文件夹

然后将可执行文件wordcount和源文件hello.txt上传到远端，命令

hadoop dfs -copyFromLocal ./wordcount /bin/

hadoop dfs -copyFromLocal ./hello.txt /home

在远端可以通过命令

hadoop dfs -ls /home

hadoop dfs -ls /bin

查看已经上传的文件

D.准备工作就绪可以跑程序了执行命令

hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -input /home/hello.txt -output /home/result -program /bin/wordcount

然后就可以到远端的/home/result中ls命令之后cat或者复制回本地了