ubuntu16.04编译hadoop-3.0.0源码

小故事:项目需要提供native lib的hadoop3.0.0。但官方的二进制文件默认只提供了内建的java版本,所以这个blog的目的实际上是说明一下编译过程中遇到的坑,同时编译出native lib并替换二进制版本中的native文件夹.
完工以后结果就是这样的:

hadoop checknative
2018-07-24 00:11:53,480 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-07-24 00:11:53,483 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
snappy:  true /usr/lib/x86_64-linux-gnu/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
ISA-L:   true /usr/lib/libisal.so.2

1.系统环境：

OS
16.04.1-Ubuntu
kernel
4.15.0-24
源码下载地址 http://archive.apache.org/dist/hadoop/core/hadoop-3.0.0/
下载源码后，解压源码，进入根目录查看 BUILDING.txt

Requirements:

* Unix System
* JDK 1.8+
* Maven 3.3 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)

2.安装jdk1.8

下载oracle官网的java8,解压后安装即可

export JAVA_HOME=/opt/jdk1.8**
export CLASSPATH=$JAVA_HOME/lib/
export PATH=$PATH:$JAVA_HOME/bin

3.安装Maven3.5.0

下载地址:http://maven.apache.org/download.cgi
配置环境变量

export MAVEN_HOME=/usr/bin/apache-maven-3.5.0 
export PATH=$PATH:%MAVEN_HOME%/bin 

验证安装有效性:

mvn -v
>>>Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /usr/bin/apache-maven-3.5.4
Java version: 1.8.0_172, vendor: Oracle Corporation, runtime: /usr/lib/jvm/jdk1.8.0_172/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "4.15.0-24-generic", arch: "amd64", family: "unix"

3.1 配置Maven

打开Maven安装目录下\conf\settings.xml文件
在settings下添加本地仓库地址，如/home/bear/maven/maven-dependcies

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <!-- localRepository
   | The path to the local repository maven will use to store artifacts.
   |
   | Default: ${user.home}/.m2/repository
  <localRepository>/path/to/local/repo</localRepository>
  -->
  <localRepository>/home/bear/maven/maven-dependcies</localRepository>

在同一个文件中profiles项配置中添加

<profile>
    <profile>
      <id>jdk-1.8</id>  
      <activation>  
          <activeByDefault>true</activeByDefault>  
          <jdk>1.8</jdk>  
      </activation>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>  
        <maven.compiler.target>1.8</maven.compiler.target>  
        <maven.compiler.compilerVersion>1.8</maven.compiler.compilerVersion>
    </properties>
    </profile>
    </profile>
    <repositories>   
        <repository>     
            <id>Ibiblio</id>     
            <name>Ibiblio</name>     
            <url>http://repo1.maven.org/maven2/</url>   
        </repository>   
        <repository>     
            <id>PlanetMirror</id>     
            <name>Planet Mirror</name>     
            <url>http://public.planetmirror.com/pub/maven/</url>   
        </repository> 
    </repositories>

4.ProtocolBuffer 2.5.0 编译安装

hadoop使用protocol buffer进行通信，需要下载和安装protobuf-2.5.0.tar.gz。
安装

./configure
make && make install

输入 protoc --version命令,判断安装是否成功.

好吧,不一定成功啦.
错误原因：
protobuf的默认安装路径是/usr/local/lib，而/usr/local/lib 不在Ubuntu默认的LD_LIBRARY_PATH里，所以就找不到该lib
解决方法：
1. 创建文件 /etc/ld.so.conf.d/libprotobuf.conf并包含内容：

/usr/local/lib  

1. 输入命令

sudo ldconfig 

这时，再运行protoc --version就可以正常看到版本号了

5.安装zstd

git clone https://github.com/facebook/zstd.git
cd zstd
make
sudo make install

6.用maven开始编译hadoop3.0.0

1. mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true 仅编译正常源码部分，对于native部分不进行编译，最终结果打包

2. mvn package -Pdist,native,docs -DskipTests -Dtar 编译正常部分源码、native依赖库以及帮助文档，最终结果打包

problem1

https://issues.apache.org/jira/browse/HADOOP-15122

problem2

hadoop-client-check-test-invariants 编译失败
这个简单粗暴,直接删除对应的文件夹和同一目录pom.xml文件中

<module>hadoop-client-check-test-invariants </module>

好吧,到这里应该已经可以看到成功的提示了

到这里,其实还差点

hadoop checknative
2018-07-23 22:26:34,829 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2018-07-23 22:26:34,831 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2018-07-23 22:26:34,834 WARN zstd.ZStandardCompressor: Error loading zstandard native libraries: java.lang.InternalError: Cannot load libzstd.so.1 (libzstd.so.1: 无法打开共享对象文件: 没有那个文件或目录)!
Native library checking:
hadoop:  true /usr/local/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  false 
snappy:  true /usr/lib/x86_64-linux-gnu/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
ISA-L:   true /usr/lib/libisal.so.2

我们发现hadoop居然没有找到libzstd.so.1的库.直接到zstd的下载目录中

BUCK      decompress   dll        libzstd.pc     libzstd.so.1      README.md
common    deprecated   legacy     libzstd.pc.in  libzstd.so.1.3.5  zstd.h
compress  dictBuilder  libzstd.a  libzstd.so     Makefile

把libzstd.so*全部copy到/lib/x86_64-linux-gnu目录下即可.