Hadoop源码本机编译

在启动hadoop时，我看到有下面警告信息

[hadoop@Yarn-Master ~]$ hadoop/2.6.1/sbin/start-dfs.sh 
15/12/14 10:19:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: starting namenode, logging to /home/hadoop/hadoop/2.6.1/logs/hadoop-hadoop-namenode-Yarn-Master.out
master: starting datanode, logging to /home/hadoop/hadoop/2.6.1/logs/hadoop-hadoop-datanode-Yarn-Master.out
slave_2: ssh: connect to host slave_2 port 22: No route to host
slave_1: ssh: connect to host slave_1 port 22: No route to host
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop/2.6.1/logs/hadoop-hadoop-secondarynamenode-Yarn-Master.out
15/12/14 10:19:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@Yarn-Master ~]$ 

Google出来可以参考 http://hadoop.apache.org/docs/r2.6.1/hadoop-project-dist/hadoop-common/NativeLibraries.html这篇官方文档需要自己编译hadoop的本地库，我查看了一下自己的本地库，果然是64位的，但我的机器是32位的

[hadoop@Yarn-Master ~]$ file hadoop/2.6.1/lib/native/libhadoop.so.1.0.0 
hadoop/2.6.1/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
[hadoop@Yarn-Master ~]$ 

自己从网上下载了2.6.1版本的源码包，直接编译，报各种错误，然后各种Google，找到两篇不错的博客：centos下 http://blog.pengduncun.com/?p=1208，ubuntu下 http://my.oschina.net/laigous/blog/356552，虽然两个系统需要的软件发布版本不同，但要装的软件是一样的，下面均以CentOS为例。

编译需要的软件

首先是官方文档上写明了的

The packages you need to install on the target platform are:

• C compiler (e.g. GNU C Compiler)
• GNU Autools Chain: autoconf, automake, libtool
• zlib-development package (stable version >= 1.2.0)
• openssl-development package(e.g. libssl-dev)

Once you installed the prerequisite packages use the standard hadoop pom.xml file and pass along the native flag to build the native hadoop library:

1、C相关的编译器，可以直接使用 yum groupinstall "Development Tools"来安装，有人说gcc需要4.4版本的，但是我的gcc默认版本就是4.4的所以没验证

[hadoop@Yarn-Master ~]$ gcc --version
gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[hadoop@Yarn-Master ~]$ 

[hadoop@Yarn-Master ~]$ libtool --version
ltmain.sh (GNU libtool) 2.2.6b
Written by Gordon Matzigkeit <gord@gnu.ai.mit.edu>, 1996

Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
[hadoop@Yarn-Master ~]$ automake --version
automake (GNU automake) 1.11.1
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Tom Tromey <tromey@redhat.com>
       and Alexandre Duret-Lutz <adl@gnu.org>.
[hadoop@Yarn-Master ~]$ autoconf --version
autoconf (GNU Autoconf) 2.63
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv2+: GNU GPL version 2 or later
<http://gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by David J. MacKenzie and Akim Demaille.
[hadoop@Yarn-Master ~]$ 

2、zlib zlib-devel openssl openssl-devel这四个库可以直接使用yum安装

[root@Yarn-Master hadoop]# yum install zlib zlib-devel openssl openssl-devel
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirrors.zju.edu.cn
 * extras: mirrors.zju.edu.cn
 * updates: mirrors.zju.edu.cn
Package zlib-1.2.3-29.el6.i686 already installed and latest version
Package zlib-devel-1.2.3-29.el6.i686 already installed and latest version
Package openssl-1.0.1e-42.el6.i686 already installed and latest version
Package openssl-devel-1.0.1e-42.el6.i686 already installed and latest version
Nothing to do
[root@Yarn-Master hadoop]# 

常规的java编译需要的软件

maven： http://maven.apache.org/download.cgi，ant： http://ant.apache.org/bindownload.cgi下载软件包以后，解压到/usr/local文件夹下，

[root@Yarn-Master hadoop]# ls -al /usr/local/
total 64
drwxr-xr-x. 16 root root 4096 Dec 13 16:28 .
drwxr-xr-x. 13 root root 4096 Oct 28 09:55 ..
drwxr-xr-x.  6 root root 4096 Dec 13 16:15 apache-ant-1.9.6
drwxr-xr-x.  6 root root 4096 Dec 11 17:49 apache-maven-3.3.9
drwxr-xr-x.  2 root root 4096 Sep 23  2011 bin
drwxr-xr-x.  2 root root 4096 Sep 23  2011 etc
drwxr-xr-x.  8 root root 4096 Dec 13 16:28 findbugs-3.0.1
drwxr-xr-x.  2 root root 4096 Sep 23  2011 games
drwxr-xr-x.  2 root root 4096 Sep 23  2011 include
drwxr-xr-x.  2 root root 4096 Sep 23  2011 lib
drwxr-xr-x.  2 root root 4096 Sep 23  2011 libexec
drwxr-xr-x.  5 root root 4096 Dec 11 19:56 protobuf-2.5.0
drwxr-xr-x.  5 root root 4096 Dec 11 19:06 protobuf-2.6.1
drwxr-xr-x.  2 root root 4096 Sep 23  2011 sbin
drwxr-xr-x.  5 root root 4096 Oct 28 09:21 share
drwxr-xr-x.  2 root root 4096 Sep 23  2011 src
[root@Yarn-Master hadoop]# 

然后，再在/etc/profile.d/下建2个shell文件：ant.sh, maven.sh，其中MAVEN的shell文件如下：

[root@Yarn-Master hadoop]# cat /etc/profile.d/maven.sh 
#
# set the maven environment
#
export MAVEN_HOME=/usr/local/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$PATH
[root@Yarn-Master hadoop]# 

然后，运行（ 该命令只在当前用户下生效，如果切换用户需要再执行下，但是重启后会自动执行的）

[root@Yarn-Master hadoop]# source /etc/profile

查看是否安装正确

[root@Yarn-Master hadoop]# which ant
/usr/local/apache-ant-1.9.6/bin/ant
[root@Yarn-Master hadoop]# which mvn
/usr/local/apache-maven-3.3.9/bin/mvn
[root@Yarn-Master hadoop]# 

其他hadoop需要的

1、cmake：可以直接使用yum命令安装

[root@Yarn-Master hadoop]# yum install cmake
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirrors.zju.edu.cn
 * extras: mirrors.zju.edu.cn
 * updates: mirrors.zju.edu.cn
Package cmake-2.8.12.2-4.el6.i686 already installed and latest version
Nothing to do
[root@Yarn-Master hadoop]# 

2、protobuf：需要先检查下自己的hadoop编译时的版本

[hadoop@Yarn-Master ~]$ hadoop/2.6.1/bin/hadoop version
Hadoop 2.6.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b4d876d837b830405ccdb6af94742f99d49f9c04
Compiled by jenkins on 2015-09-16T21:07Z
Compiled with protoc 2.5.0
From source with checksum ba9a9397365e3ec2f1b3691b52627f
This command was run using /home/hadoop/hadoop/2.6.1/share/hadoop/common/hadoop-common-2.6.1.jar
[hadoop@Yarn-Master ~]$ 

然后去 https://github.com/google/protobuf/tags找到对应的版本，下载安装就好

3、findbugs：http://findbugs.sourceforge.net/downloads.html同上

[hadoop@Yarn-Master ~]$ which protoc
/usr/local/protobuf-2.5.0/bin/protoc
[hadoop@Yarn-Master ~]$ which fb
/usr/local/findbugs-3.0.1/bin/fb
[hadoop@Yarn-Master ~]$ 

编译

完成以上步骤后，就可以使用官方文档里的命令完成编译工作

Once you installed the prerequisite packages use the standard hadoop pom.xml file and pass along the native flag to build the native hadoop library:

   $ mvn package -Pdist,native -DskipTests -Dtar

You should see the newly-built library in:

   $ hadoop-dist/target/hadoop-2.6.1/lib/native

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  2.740 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.573 s]
...
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.524 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:16 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:54 min
[INFO] Finished at: 2015-12-13T18:52:17+08:00
[INFO] Final Memory: 86M/247M
[INFO] ------------------------------------------------------------------------
[hadoop@Yarn-Master hadoop-2.6.1-src]$ 

检查是否编译正确

[hadoop@Yarn-Master ~]$ file downloads/hadoop-2.6.1-src/hadoop-dist/target/hadoop-2.6.1/lib/native/libhadoop.so.1.0.0 
downloads/hadoop-2.6.1-src/hadoop-dist/target/hadoop-2.6.1/lib/native/libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
[hadoop@Yarn-Master ~]$ 

使用编译的本地库

先将编译后的库拷贝到hadoop目录中

[hadoop@Yarn-Master ~]$ cp -r downloads/hadoop-2.6.1-src/hadoop-dist/target/hadoop-2.6.1/lib hadoop/src_2.6.1/
[hadoop@Yarn-Master ~]$ ls hadoop/src_2.6.1/
lib
[hadoop@Yarn-Master ~]$ ls hadoop/src_2.6.1/lib/native/
libhadoop.a       libhadoop.so        libhadooputils.a  libhdfs.so
libhadooppipes.a  libhadoop.so.1.0.0  libhdfs.a         libhdfs.so.0.0.0
[hadoop@Yarn-Master ~]$ 

然后修改环境变量，使用编译的本地库

export HADOOP_SRC_HOME=$HOME/hadoop/src_2.6.1

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_SRC_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_SRC_HOME/lib/native"

再启动就可以看到警告信息消失了

[hadoop@Yarn-Master ~]$ export HADOOP_ROOT_LOGGER=DEBUG,console
[hadoop@Yarn-Master ~]$ hadoop/2.6.1/sbin/start-dfs.sh 
15/12/14 11:25:44 DEBUG util.Shell: setsid exited with exit code 0
...
15/12/14 11:25:45 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
15/12/14 11:25:45 DEBUG security.Groups:  Creating new Groups object
15/12/14 11:25:45 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
15/12/14 11:25:45 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
15/12/14 11:25:45 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
15/12/14 11:25:45 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping