Hadoop在CentOS 6.6-x86_64上编译

写在前面的话：

hadoop默认下载镜像里面的提供两种选择：源码包，编译好的。比如猫哥为了图方便，从下面链接里直接找了一个编译好的。

http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/

下载、配置一路还算可以，只是put什么的就出问题了。报错内容：

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

查了查，貌似是32位、64位问题。就是说，官网上提供的是32位机上编译好的。64位系统的话，需要自己编译。

【环境】

CentOS 6.6

jdk-1.7.0-x86_64

【准备工作】

1. 安装一系列软件包：

执行如下命令：

yum install autoconfautomake libtool cmake ncurses-devel openssl-devel gcc* 

2. 安装Maven

2.1 直接官网下载一个最新版编译好的，比如：http://mirrors.hust.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

2.2 解压缩，放到随便哪个位置。

tar xf apache-maven-3.3.9-bin.tar.gz
mv apache-maven-3.3.9 /data/maven339

2.3 编辑环境变量：vi /etc/profile，在最下面添加如下内容：

export MAVEN_HOME=/letv/maven339
export PATH=$PATH:$MAVEN_HOME/bin:

source /etc/profile使之生效。

2.4 验证：执行如下命令，有结果则说明maven装好了。

mvn --version

3. 安装Protocbuf

记得Protocbuf必须是2.5.0版本。本来猫哥找到一个最新版的，结果编译不过去，查看报错原因，说不是“2.5.0”。也是醉了。搜索一下哪里有这个版本吧。

3.1 安装很简单，依次执行：

./configure --prefix=/usr/local/protoc
make
make check
make install

3.2 编辑环境变量，vi /etc/profile，使之生效：

最懒的办法，直接在最后添加PATH的地方补一段，记得冒号“：”分开就可以了。

export PATH=$PATH:$MAVEN_HOME/bin:/usr/local/protoc/bin

3.3 让环境变量生效

source /etc/profile

4. 编译hadoop

4.1 下载源码包：
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/
上面链接，找带“src”那个就可以。

4.2 编译

tar xf hadoop-2.7.2-src.tar.gz
cd hadoop-2.7.2-src
mvn package -Pdist,native -DskipTests -Dtar

mvn过程会很长，20分钟算很幸运的！反正猫哥这么一行命令敲下去，再没报错。

编译好的结果看起来很酷：

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.116 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.715 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.838 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.128 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.521 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.044 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.604 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  2.623 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.420 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:34 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  4.375 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 14.680 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.024 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:30 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 18.526 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  5.891 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.811 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.026 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.023 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [01:34 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 30.626 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.026 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  8.511 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 14.287 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.476 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  5.793 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 15.043 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  4.242 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  5.157 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  2.664 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.038 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  1.903 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.328 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.021 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  4.723 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  3.674 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.910 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 15.041 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 13.080 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  3.921 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  6.833 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  4.313 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  6.208 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  4.336 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  5.000 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  3.334 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  4.060 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  7.985 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.673 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  4.210 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.296 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.966 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.728 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.245 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  8.479 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.518 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 10.013 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  5.229 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  7.260 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.053 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.415 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  7.485 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.038 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 31.595 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:09 min
[INFO] Finished at: 2017-03-15T16:50:46+08:00
[INFO] Final Memory: 213M/1044M
[INFO] ------------------------------------------------------------------------

5. 采摘结果

编译结果应该是在 hadoop-2.7.2-src/hadoop-dist/target/hadoop-2.7.2/。

剩下的就是Hadoop配置问题了。