本文主要记录Hadoop2.x源码编译的详细步骤:
1.编译基础环境
- CentOS 6.7 minimal
- Hadoop 2.7.4
- JDK 1.7+
- Maven 3.0 or later
- Findbugs 1.3.9 (if running findbugs)
- ProtocolBuffer 2.5.0
- CMake 2.6 or newer (if compiling native code)
- Zlib devel (if compiling native code)
- openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
- Internet connection for first build
系统原始环境特别说明:
系统选择minimal最小系统安装,并自定义选装以下包: Base System中的Base、Compatbillty libraries、Debugging Tools; Development中的 Development tools
2.yum 源配置
参考本博客的yum源配置的文章:yum源配置为阿里云镜像站
3.下载Hadoop源码包
[root@hadoop001 sourcecode]# pwd
/opt/sourcecode
[root@hadoop001 sourcecode]# mkdir -p /opt/sourcecode
[root@hadoop001 sourcecode]#wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4-src.tar.gz
[root@hadoop001 sourcecode]#tar -xzvf hadoop-2.7.4-src.tar.gz
[root@hadoop001 sourcecode]#cat ./hadoop-2.7.4-src/BUILDING.txt
#从BUILDING文件中我们可以看到编译的要求
4.JDK安装
参考本博客的JDK安装的文章:JDK安装
5.Maven安装
mkdir -p /opt/software
cd /opt/software
上传apache-maven-3.3.9-bin.zip
unzip apache-maven-3.3.9-bin.zip
#添加maven环境变量
echo '#SET MAVEN'>>/etc/profile
echo 'export MAVEN_HOME=/opt/software/apache-maven-3.3.9' >>/etc/profile
echo 'export MAVEN_OPTS="-Xms256m -Xmx512m"'>>/etc/profile
echo 'export PATH=$MAVEN_HOME/bin:$PATH'>>/etc/profile
source /etc/profile
mvn --version
6.protobuf安装
protobuf要编译安装,需安装gcc、gcc-c++、 make
上传 protobuf-2.5.0.tar.gz
tar -xzvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
yum install -y gcc gcc-c++ make
./configure --prefix=/usr/local/protobuf
make && make install
#添加protobuf环境变量
echo '#SET PROTOBUF'>>/etc/profile
echo 'export PROTOC_HOME=/usr/local/protobuf'>>/etc/profile
echo 'export PATH=$PROTOC_HOME/bin:$PATH'>>/etc/profile
source /etc/profile
protoc --version
7.Findbugs安装
上传 findbugs-1.3.9.zip
unzip findbugs-1.3.9.zip
#添加Findbugs环境变量
echo '#SET Findbugs'>>/etc/profile
echo 'export FINDBUGS_HOME=/opt/software/findbugs-1.3.9'>>/etc/profile
echo 'export PATH=$FINDBUGS_HOME/bin:$PATH'>>/etc/profile
8.Snappy压缩库安装
安装snappy1.1.4,使Hadoop支持snappy压缩
wget https://github.com/google/snappy/releases/download/1.1.4/snappy-1.1.4.tar.gz
tar -zxvf snappy-1.1.4.tar.gz
cd snappy-1.1.4
./configure
make && make install
ll -h /usr/local/lib |grep snappy
9.其他依赖安装
yum install -y ant openssl openssl-devel svn ncurses-devel zlib-devel libtool svn
yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake
10.编译
进入Hadooop源码目录
cd hadoop-2.7.4-src
mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy
11.生成tar包
/opt/sourcecode/hadoop-2.7.4-src/hadoop-dist/target/hadoop-2.7.4.tar.gz
注意事项:
- 由于Maven仓库在墙外,Maven在编译项目时下载包卡住情况,ctrl+c 中断,重新执行编译。
- 如果出现提示缺少了某个文件的情况,则要先清理maven(使用命令 mvn clean) 再重新编译。