编译hadoop，支持多种压缩格式

环境准备

1、hadoop伪分布式：hadoop-2.6.0-cdh5.7.0
2、软件环境
链接：https://pan.baidu.com/s/1t52hB8f1t3bHL39iFmt_hg
提取码：t97j

编译hadoop

1. 安装依赖库

[root@hadoop001 ~]# yum install -y svn ncurses-devel
[root@hadoop001 ~]# yum install -y gcc gcc-c++ make cmake
[root@hadoop001 ~]# yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
[root@hadoop001 ~]# yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake cmake 

2. 上传软件包

[hadoop@hadoop001 ~]$ mkdir opt
# 将软件包上传到opt目录
[hadoop@hadoop001 ~]$ cd opt/
[hadoop@hadoop001 ~]$ ll
-rw-r--r--. 1 hadoop hadoop   9136463 4月  18 00:56 apache-maven-3.6.1-bin.tar.gz
-rw-r--r--. 1 hadoop hadoop  42610549 4月  18 00:57 hadoop-2.6.0-cdh5.7.0-src.tar.gz
-rw-rw-r--. 1 hadoop hadoop 191963015 4月  17 07:58 hadoop-2.6.0-cdh5.7.0.tar.gz
-rw-r--r--. 1 hadoop hadoop 138315801 4月  18 00:56 jdk1.7.0_45.tar.gz
-rw-r--r--. 1 hadoop hadoop   2401901 4月  18 00:56 protobuf-2.5.0.tar.gz

3. 安装JDK1.7， JDK1.8会编译失败

[root@hadoop614 ~]# tar -zxvf /home/hadoop/opt/jdk1.7.0_45.tar.gz /usr/java/
[root@hadoop614 ~]# cd /usr/java/
[root@hadoop614 java]# ll
总用量 8
drwxr-xr-x. 8 root root 4096 2月   8 2014 jdk1.7.0_45
drwxr-xr-x. 8 root root 4096 4月  11 2015 jdk1.8.0_45

# 修改Java环境变量
# env
#export JAVA_HOME=/usr/java/jdk1.8.0_45
export JAVA_HOME=/usr/java/jdk1.7.0_45
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/lib:$PATH

4. 安装maven

[hadoop@hadoop614 app]$ tar -zxvf ../opt/apache-maven-3.6.1-bin.tar.gz -C ./
[hadoop@hadoop614 app]$ ln -s apache-maven-3.6.1/ maven

• 配置maven环境变量

export MAVEN_HOME=/home/hadoop/app/maven
export MAVEN_OPTS="-Xms1024m -Xmx1024m"
export PATH=$MAVEN_HOME/bin:$PATH

• 配置maven仓库

[hadoop@hadoop614 app]$ vi maven/conf/settings.xml

#配置maven的本地仓库位置
<localRepository>/home/hadoop/app/maven/repo</localRepository>
# 在<mirrors>  </mirrors>标签中添加添加阿里云中央仓库地址
<mirror>
       <id>nexus-aliyun</id>
       <mirrorOf>central</mirrorOf>
       <name>Nexus aliyun</name>
       <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

5. 解压repo.tar.gz

[hadoop@hadoop614 maven]$ tar -zxvf ~/opt/repo.tar.gz -C ~/app/maven/
[hadoop@hadoop614 maven]$ cd ~/app/maven/
[hadoop@hadoop614 maven]$ ll
总用量 44
drwxrwxr-x.  2 hadoop hadoop  4096 4月  18 01:03 bin
drwxrwxr-x.  2 hadoop hadoop  4096 4月  18 01:03 boot
drwxr-xr-x.  3 hadoop hadoop  4096 4月  18 01:09 conf
drwxr-xr-x.  4 hadoop hadoop  4096 4月  18 01:03 lib
-rw-r--r--.  1 hadoop hadoop 13437 4月   5 03:02 LICENSE
-rw-r--r--.  1 hadoop hadoop   182 4月   5 03:02 NOTICE
-rw-r--r--.  1 hadoop hadoop  2533 4月   5 02:56 README.txt
drwxrwxr-x. 52 hadoop hadoop  4096 12月  1 14:58 repo

6. 安装protobuf

[hadoop@hadoop614 app]$ tar -zxvf ~/opt/protobuf-2.5.0.tar.gz -C ./
[hadoop@hadoop614 app]$ cd protobuf-2.5.0/
[hadoop@hadoop001 protobuf-2.5.0]$ ./configure  --prefix=/home/hadoop/app/protobuf-2.5.0
#编译以及安装
[hadoop@hadoop001 protobuf-2.5.0]$ make && make install

• 添加环境变量

export PROTOBUF_HOME=/home/hadoop/app/protobuf-2.5.0
export PATH=$PROTOBUF_HOME/bin:$PATH

7. 编译hadoop

• 解压编译包

[hadoop@hadoop614 ~]$ tar -zxvf opt/hadoop-2.6.0-cdh5.7.0-src.tar.gz -C source/
[hadoop@hadoop614 ~]$ cd ~/source/hadoop-2.6.0-cdh5.7.0/
#进行编译，第一次编译会下载很多依赖的jar包，快慢由网速决定，需耐心等待
[hadoop@hadoop614 hadoop-2.6.0-cdh5.7.0]$ mvn clean package -Pdist,native -DskipTests -Dtar

• 查看编译后的包：hadoop-2.6.0-cdh5.7.0.tar.gz

#有 BUILD SUCCESS 信息则表示编译成功
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 13.592 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 12.042 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.094 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:49 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 37:39 min
[INFO] Finished at: 2019-04-07T16:48:42+08:00
[INFO] Final Memory: 200M/989M
[INFO] ------------------------------------------------------------------------
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ ll /home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/
total 564036
drwxrwxr-x. 2 hadoop hadoop      4096 Apr  7 16:46 antrun
drwxrwxr-x. 3 hadoop hadoop      4096 Apr  7 16:46 classes
-rw-rw-r--. 1 hadoop hadoop      1998 Apr  7 16:46 dist-layout-stitching.sh
-rw-rw-r--. 1 hadoop hadoop       690 Apr  7 16:47 dist-tar-stitching.sh
drwxrwxr-x. 9 hadoop hadoop      4096 Apr  7 16:47 hadoop-2.6.0-cdh5.7.0
-rw-rw-r--. 1 hadoop hadoop 191880143 Apr  7 16:47 hadoop-2.6.0-cdh5.7.0.tar.gz
-rw-rw-r--. 1 hadoop hadoop      7314 Apr  7 16:47 hadoop-dist-2.6.0-cdh5.7.0.jar
-rw-rw-r--. 1 hadoop hadoop 385618309 Apr  7 16:48 hadoop-dist-2.6.0-cdh5.7.0-javadoc.jar
-rw-rw-r--. 1 hadoop hadoop      4855 Apr  7 16:47 hadoop-dist-2.6.0-cdh5.7.0-sources.jar
-rw-rw-r--. 1 hadoop hadoop      4855 Apr  7 16:47 hadoop-dist-2.6.0-cdh5.7.0-test-sources.jar
drwxrwxr-x. 2 hadoop hadoop      4096 Apr  7 16:47 javadoc-bundle-options
drwxrwxr-x. 2 hadoop hadoop      4096 Apr  7 16:47 maven-archiver
drwxrwxr-x. 3 hadoop hadoop      4096 Apr  7 16:46 maven-shared-archive-resources
drwxrwxr-x. 3 hadoop hadoop      4096 Apr  7 16:46 test-classes
drwxrwxr-x. 2 hadoop hadoop      4096 Apr  7 16:46 test-dir

• 使用编译好的hadoop-2.6.0-cdh5.7.0.tar.gz包，替换掉原来的伪分布式部署的目录

验证结果

[hadoop@hadoop614 ~]$ hadoop checknative
19/04/19 04:19:22 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
19/04/19 04:19:22 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/lib64/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so