hadoop2.5.0源码编译

hadoop2.5.0源码编译

标签（空格分隔）： hadoop2.5.0 centOS Maven Protobuf JDK 源码编译

1.编译软硬件基本环境

1.1、硬件环境：

操作系统：CentOS 6.4版本64位

1.2、软件环境：

hadoop：    2.5.0
JDK：       1.7.0
maven：     3.0.5
protobuf：  2.5.0
findbugs：  1.3.9

到相应官网下载好相应软件，并且解压到linux相应目录
hadoop:https://archive.apache.org/dist/hadoop/common/hadoop-2.5.0/
JDK1:http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
maven:http://maven.apache.org/docs/3.0.5/release-notes.html
protobuf:http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
findbugs:http://sourceforge.net/projects/findbugs/

2.确认linux网络正常

设置为网络连接方式为“Nat连接”，设置自动获取IP。

验证方法:
在终端输入命令:ping www.baidu.com
[xuxu@bigdata01 ~]$ ping www.baidu.com
PING www.a.shifen.com (58.217.200.37) 56(84) bytes of data.
64 bytes from 58.217.200.37: icmp_seq=1 ttl=128 time=63.0 ms
64 bytes from 58.217.200.37: icmp_seq=2 ttl=128 time=60.8 ms
64 bytes from 58.217.200.37: icmp_seq=3 ttl=128 time=64.4 ms
64 bytes from 58.217.200.37: icmp_seq=4 ttl=128 time=62.0 ms

3.停止hadoop守护进程

比如namenode/datanode/resourcemanager/nodemanager...

4.linux下安装JDK

 JDK解压后，只需配置环境变量即可。
 命令：$sudo vi /etc/profile
在该文件末尾加上下面两句

export JAVA_HOME=/opt/modules/jdk1.7.0_67 //JDK解压的目录
export PATH=$PATH:$JAVA_HOME/bin

<验证是否成功>
命令：source /etc/profile //每次配置环境变量都需要source
命令：java -version
显示如下，就证明已成功！

java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

5.安装maven

maven解压后，只需配置环境变量即可。

 命令：$sudo vi /etc/profile
 在该文件末尾加上下面两句

 export MAVEN_HOME=/opt/modules/apache-maven-3.0.5  //maven解压目录
 export PATH=.:$MAVEN_HOME/bin:$PATH

<验证是否成功>

命令：source /etc/profile //每次配置环境变量都需要source
命令：java -version
有如下信息，就证明已成功！

Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 08:51:28-0500)
Maven home: /opt/modules/apache-maven-3.0.5
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: /opt/modules/jdk1.7.0_67/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch:     "amd64", family: "unix"

6.安装protobuf

安装Protobuf，其安装需要对Protobuf进行编译，故我们需要编译的依赖环境gcc、gcc-c++、cmake、openssl-devel、ncurses-devel，安装命令如下所示：

sudo yum -y install gcc   
sudo yum -y install gcc-c++
sudo yum -y install cmake
sudo yum -y install openssl-devel
sudo yum -y install ncurses-devel

下面一步步验证这些编译环境是否安装成功.

6.1、gcc成功安装验证信息如下

[xuxu@bigdata01 modules]$ sudo gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr                 --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) 

6.2、cmake安装成功信息如下：

[xuxu@bigdata01 modules]$ sudo make -v
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying     conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-redhat-linux-gnu

6.3、现在开始去编译Protobuf，编译命令如下所示：

 [xuxu@bigdata01 protobuf-2.5.0]$ sudo ./configure --prefix=/usr/local/protoc

[xuxu@bigdata01 protobuf-2.5.0]$sudo make [xuxu@bigdata01 protobuf-2.5.0]$ sudo make install

6.4、验证protobuf是否安装成功

[xuxu@bigdata01 protobuf-2.5.0]$ protoc --version

libprotoc 2.5.0
[xuxu@bigdata01 protobuf-2.5.0]$

7.现在开始编译hadoop2.5.0

在编译hadoop源码之前，首先应该先配置下以下信息

[xuxu@bigdata01 modules]$ sudo vi apache-maven-3.0.5/conf/settings.xml 
在该文件的
<mirrors>处增加以下内容
<mirror>
  <id>nexus-osc</id>
  <mirrorOf>*</mirrorOf>
  <name>Nexus osc</name>
  <url>http://maven.oschina.net/content/groups/public/</url>
</mirror>

这是配置国内镜像用的，增加这个可以让编译源码的时候不用到国外的服务器上下载jar包，减少编译时间。

---

然后进入hadoop源码目录，进行编译…

[xuxu@bigdata01 modules]$ cd hadoop-2.5.0-src/

[xuxu@bigdata01 hadoop-2.5.0-src]$mvn package -DskipTests -Pdist,native 在编译成功后，会出现以下信息： [INFO] hadoop-yarn-applications …………………….. SUCCESS [0.195s] [INFO] hadoop-yarn-applications-distributedshell ……… SUCCESS [5.060s] [INFO] hadoop-yarn-applications-unmanaged-am-launcher …. SUCCESS [2.508s] [INFO] hadoop-yarn-site ……………………………. SUCCESS [0.079s] [INFO] hadoop-yarn-project …………………………. SUCCESS [4.953s] [INFO] hadoop-mapreduce-client ……………………… SUCCESS [0.092s] [INFO] hadoop-mapreduce-client-core …………………. SUCCESS [32.330s] [INFO] hadoop-mapreduce-client-common ……………….. SUCCESS [24.894s] [INFO] hadoop-mapreduce-client-shuffle ………………. SUCCESS [5.300s] [INFO] hadoop-mapreduce-client-app ………………….. SUCCESS [14.665s] [INFO] hadoop-mapreduce-client-hs …………………… SUCCESS [11.212s] [INFO] hadoop-mapreduce-client-jobclient …………….. SUCCESS [10.009s] [INFO] hadoop-mapreduce-client-hs-plugins ……………. SUCCESS [2.140s] [INFO] Apache Hadoop MapReduce Examples ……………… SUCCESS [8.650s] [INFO] hadoop-mapreduce ……………………………. SUCCESS [3.737s] [INFO] Apache Hadoop MapReduce Streaming …………….. SUCCESS [6.551s] [INFO] Apache Hadoop Distributed Copy ……………….. SUCCESS [17.672s] [INFO] Apache Hadoop Archives ………………………. SUCCESS [2.332s] [INFO] Apache Hadoop Rumen …………………………. SUCCESS [11.085s] [INFO] Apache Hadoop Gridmix ……………………….. SUCCESS [5.454s] [INFO] Apache Hadoop Data Join ……………………… SUCCESS [3.467s] [INFO] Apache Hadoop Extras ………………………… SUCCESS [3.664s] [INFO] Apache Hadoop Pipes …………………………. SUCCESS [10.255s] [INFO] Apache Hadoop OpenStack support ………………. SUCCESS [7.210s] [INFO] Apache Hadoop Client ………………………… SUCCESS [4.480s] [INFO] Apache Hadoop Mini-Cluster …………………… SUCCESS [0.213s] [INFO] Apache Hadoop Scheduler Load Simulator ………… SUCCESS [7.422s] [INFO] Apache Hadoop Tools Dist …………………….. SUCCESS [3.368s] [INFO] Apache Hadoop Tools …………………………. SUCCESS [0.171s] [INFO] Apache Hadoop Distribution …………………… SUCCESS [18.150s] [INFO] ———————————————————————— [INFO] BUILD SUCCESS [INFO] ———————————————————————— [INFO] Total time: 26:47.448s [INFO] Finished at: Thu Nov 05 08:28:39 EST 2015 [INFO] Final Memory: 129M/454M [INFO] ———————————————————————— [xuxu@bigdata01 hadoop-2.5.0-src]$

严格按照以上步骤配置并且每一步都验证成功后，如果还出现编译failure的情况，可能是网络问题，请重新再执行一次编译命令。有时候需要重新编译七八次才成功都有可能。当然，你也要看编译log信息。

以上都是我一步步自己验证过的，如有不对之处，请指正。