Hadoop 源码编译安装

本文主要记录Hadoop2.x源码编译的详细步骤：

1.编译基础环境

• CentOS 6.7 minimal
• Hadoop 2.7.4
• JDK 1.7+
• Maven 3.0 or later
• Findbugs 1.3.9 (if running findbugs)
• ProtocolBuffer 2.5.0
• CMake 2.6 or newer (if compiling native code)
• Zlib devel (if compiling native code)
• openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
• Internet connection for first build

系统原始环境特别说明：
系统选择minimal最小系统安装，并自定义选装以下包： Base System中的Base、Compatbillty libraries、Debugging Tools; Development中的 Development tools

2.yum 源配置

参考本博客的yum源配置的文章:yum源配置为阿里云镜像站

3.下载Hadoop源码包

[root@hadoop001 sourcecode]# pwd
/opt/sourcecode
[root@hadoop001 sourcecode]# mkdir -p /opt/sourcecode
[root@hadoop001 sourcecode]#wget  http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4-src.tar.gz
[root@hadoop001 sourcecode]#tar -xzvf  hadoop-2.7.4-src.tar.gz
[root@hadoop001 sourcecode]#cat ./hadoop-2.7.4-src/BUILDING.txt
#从BUILDING文件中我们可以看到编译的要求

4.JDK安装

参考本博客的JDK安装的文章:JDK安装

5.Maven安装

mkdir -p /opt/software
cd /opt/software
上传apache-maven-3.3.9-bin.zip
unzip apache-maven-3.3.9-bin.zip
#添加maven环境变量
echo '#SET MAVEN'>>/etc/profile
echo 'export MAVEN_HOME=/opt/software/apache-maven-3.3.9' >>/etc/profile
echo 'export  MAVEN_OPTS="-Xms256m -Xmx512m"'>>/etc/profile
echo 'export PATH=$MAVEN_HOME/bin:$PATH'>>/etc/profile
source /etc/profile
mvn --version

6.protobuf安装

protobuf要编译安装，需安装gcc、gcc-c++、 make
上传 protobuf-2.5.0.tar.gz

tar -xzvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
yum install -y gcc gcc-c++ make
./configure --prefix=/usr/local/protobuf
make && make install
#添加protobuf环境变量
echo '#SET PROTOBUF'>>/etc/profile
echo 'export PROTOC_HOME=/usr/local/protobuf'>>/etc/profile
echo 'export PATH=$PROTOC_HOME/bin:$PATH'>>/etc/profile
source /etc/profile
protoc --version

7.Findbugs安装

上传 findbugs-1.3.9.zip

unzip findbugs-1.3.9.zip
#添加Findbugs环境变量
echo '#SET Findbugs'>>/etc/profile
echo 'export  FINDBUGS_HOME=/opt/software/findbugs-1.3.9'>>/etc/profile
echo 'export PATH=$FINDBUGS_HOME/bin:$PATH'>>/etc/profile

8.Snappy压缩库安装

安装snappy1.1.4,使Hadoop支持snappy压缩

 wget https://github.com/google/snappy/releases/download/1.1.4/snappy-1.1.4.tar.gz
 tar -zxvf snappy-1.1.4.tar.gz
 cd snappy-1.1.4
 ./configure
 make && make install
 ll -h /usr/local/lib |grep snappy

9.其他依赖安装

yum install -y ant openssl openssl-devel svn ncurses-devel zlib-devel libtool svn
yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

10.编译

进入Hadooop源码目录

cd hadoop-2.7.4-src
mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy

11.生成tar包

/opt/sourcecode/hadoop-2.7.4-src/hadoop-dist/target/hadoop-2.7.4.tar.gz

注意事项：

• 由于Maven仓库在墙外，Maven在编译项目时下载包卡住情况，ctrl+c 中断，重新执行编译。
• 如果出现提示缺少了某个文件的情况，则要先清理maven(使用命令 mvn clean) 再重新编译。