京东云上，centos6.9环境下编译 hadoop-2.6.0-cdh5.15.1，以及遇到的坑

为什么要编译 Hadoop
官方提供的Hadoop bin包是不支持一些类似snappy，zlib等本地库功能，所以我们需要重新编译使Hadoop支持这些功能
1. 编译前的准备
在 hadoop-2.6.0-cdh5.15.1-src.tar.gz 中有一个 BUILDING.txt 文件，里面说明了我们编译需要准备哪些东西，以及编译命令

Requirements:

* Windows System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK or Visual Studio 2010 Professional
* Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

准备安装依赖文件，其中 JDK 必须为 1.7 版本，protobuf 必须为 2.5.0 版本，否则会报错。如果想使用后面下载完的 maven 仓库则 maven 的版本必须是 3.0 以上
apache-maven-3.3.9-bin.tar.gz 下载地址： http://apache.fayea.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
hadoop-2.6.0-cdh5.15.1-src.tar.gz 下载地址： http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.15.1-src.tar.gz
jdk-7u80-linux-x64.tar.gz  下载地址 ：https://download.oracle.com/otn/java/jdk/7u80-b15/jdk-7u80-linux-x64.tar.gz
protobuf-2.5.0.tar.gz 下载地址 由于现在protobuf-2.5.0.tar.gz已经无法在官网https://code.google.com/p/protobuf/downloads/list中下载，可在 https://github.com/protocolbuffers/protobuf/releases 找到相应的版本

2. 安装依赖库
在root用户下安装如下依赖

[root@hadoop000 ~]# yum install -y svn ncurses-devel
[root@hadoop000 ~]# yum install -y gcc gcc-c++ make cmake
[root@hadoop000 ~]# yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
[root@hadoop000 ~]# yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake cmake

3. 上传软件到云服务器
此次编译用hadoop用户，若hadoop用户不存在需要创建一下

[hadoop@hadoop000 app]$ ll
total 46912
drwxrwxr-x  6 hadoop hadoop     4096 Aug 21 20:18 apache-maven-3.6.1
drwxr-xr-x 40 hadoop hadoop     4096 Nov 29  2018 flink-1.7.0
lrwxrwxrwx  1 hadoop hadoop       23 Aug 21 18:38 hadoop -> hadoop-2.6.0-cdh5.15.1/
drwxrwxr-x 11 hadoop hadoop     4096 Aug 21 19:07 hadoop-2.6.0-cdh5.15.1
-rw-r--r--  1 root   root   48019244 Aug 27 16:55 hadoop-2.6.0-cdh5.15.1-src.tar.gz
lrwxrwxrwx  1 hadoop hadoop       16 Aug 21 18:38 zookeeper -> zookeeper-3.4.6/
drwxr-xr-x 10 hadoop hadoop     4096 Feb 20  2014 zookeeper-3.4.6

4. 安装JDK并配置环境变量
root用户：
解压安装包，安装目录必须是/usr/java，安装后记得修改拥有者为root和root用户组
添加环境变量

[root@hadoop000 jdk1.7.0_80]# vim /etc/profile 
#添加如下两行环境变量
export JAVA_HOME=/usr/java/jdk1.7.0_80
export PATH=$JAVA_HOME/bin:$PATH
[root@hadoop000 jdk1.7.0_80]# source /etc/profile
#测试java是否安装成功
[root@hadoop000 jdk1.7.0_80]# java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
[root@hadoop000 usr]# cd java
[root@hadoop000 java]# ll
total 8
drwxr-xr-x 8 uucp  143 4096 Apr 11  2015 jdk1.7.0_80
drwxr-xr-x 8 root root 4096 Apr 11  2015 jdk1.8.0_45
[root@hadoop000 java]# chown -R root:root jdk1.7.0_80/
[root@hadoop000 java]# ll
total 8
drwxr-xr-x 8 root root 4096 Apr 11  2015 jdk1.7.0_80
drwxr-xr-x 8 root root 4096 Apr 11  2015 jdk1.8.0_45
[root@hadoop000 java]# 

5. 安装maven并配置环境变量

#修改haoop用户的环境变量
[hadoop@hadoop000 ~]$ vim ~/.bash_profile
export JAVA_HOME=/usr/java/jdk1.7.0_80
#添加或修改如下内容，注意MAVEN_OPTS设置了maven运行的内存，防止内存太小导致编译失败
export MAVEN_HOME=/home/hadoop/app/apache-maven-3.3.9
export MAVEN_OPTS="-Xms1024m -Xmx1024m"
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[hadoop@hadoop000 ~]$ source ~/.bash_profile
[hadoop@hadoop000 ~]$ which mvn
~/app/apache-maven-3.3.9/bin/mvn

配置mavensetting.xml的仓库地址

[hadoop@hadoop000 ~]$ vim ~/app/apache-maven-3.3.9/conf/settings.xml
#配置maven的本地仓库位置
<localRepository>/home/hadoop/maven_repo/repo</localRepository>
#添加阿里云中央仓库地址，
<mirror>
     <id>nexus-aliyun</id>
     <mirrorOf>central</mirrorOf>
     <name>Nexus aliyun</name>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

6. 安装protobuf并配置环境变量

[hadoop@hadoop000 ~]$ tar -zxvf ~/soft/protobuf-2.5.0.tar.gz -C ~/app/
## 编译软件
[hadoop@hadoop001 protobuf-2.5.0]$ cd ~/app/protobuf-2.5.0/
## --prefix= 是用来待会编译好的包放在为路径,软件要在安装的路径#make进行编译，make install进行安装
[hadoop@hadoop000 protobuf-2.5.0]$ ./configure  --prefix=/home/hadoop/app/protobuf-2.5.0
## 编译以及安装
[hadoop@hadoop000 protobuf-2.5.0]$ make
[hadoop@hadoop000 protobuf-2.5.0]$ make install
## 添加环境变量
[hadoop@hadoop000 protobuf-2.5.0]$ vim ~/.bash_profile
export HADOOP_HOME=/home/hadoop/app/hadoop
export ZOOKEEPER_HOME=/home/hadoop/app/zookeeper
export MAVEN_HOME=/home/hadoop/app/apache-maven-3.3.9
export MAVEN_OPTS="-Xms1024m -Xmx1024m"
#追加如下两行内容，未编译前是没有bin目录的
export PROTOBUF_HOME=/home/hadoop/app/protobuf-2.5.0
export PATH=$PROTOBUF_HOME/bin:$MAVEN_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$PATH

[hadoop@hadoop001 protobuf-2.5.0]$ source ~/.bash_profile 

验证编译和设置是否成功，若出现libprotoc 2.5.0则为生效
[hadoop@hadoop000 protobuf-2.5.0]$ protoc --version
libprotoc 2.5.0
7. 编译hadoop
进入目录：/home/hadoop/app/hadoop-2.6.0-cdh5.15.1，修改pom.xml文件，将下图的https换成h