Mac上构建Hadoop本地库

待解决的问题：

Mac OSX 下运行Hadoop发出警告:

19/05/23 10:33:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

环境：

• 操作系统

  $ sw_vers
  ProductName:	Mac OS X
  ProductVersion:	10.13.6
  

• Hadoop版本

  $ hadoop version
  Hadoop 2.7.1
  

• jdk版本

  $ java -version
  java version "1.8.0_172"
  

安装构建所需依赖：

brew update
brew upgrade

brew install wget gcc autoconf automake libtool cmake snappy gzip bzip2 zlib openssl protobuf@2.5

1. Hadoop 源码

#克隆源码
git clone https://github.com/apache/hadoop.git

cd hadoop
#检出需要版本
git checkout branch-2.7.1
#编译打包
mvn clean package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
#copy 编译好的库至$HADOOP_HOME
cp -R hadoop-dist/target/hadoop-2.7.1/lib $HADOOP_HOME

2. Tomcat 步骤

由于天朝网络原因，在编译过程中无法完整下载tomcat压缩包导致失败，将原有的tomcat.download.url替换为可用路径即可，如下示例:

<!--<tomcat.version>6.0.41</tomcat.version>-->
<tomcat.version>7.0.94</tomcat.version>
<tomcat.download.url>
  <!--http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz-->
  https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-7/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz
</tomcat.download.url>

可以解决编译过程中会出现的如下问题

[exec]gzip: ../../downloads/apache-tomcat-6.0.41.tar.gz: unexpected end of file
[exec] gzip: ../../downloads/apache-tomcat-6.0.41.tar.gz: uncompress failed
[exec] apache-tomcat-6.0.41/bin/commons-daemon-native.tar.gz: Truncated tar archive
[exec] tar: Error exit delayed from previous errors.

3. OpenSSL 步骤

Homebrew 默认没有添加OpenSSL的link，但是cmake编译的过程需要使用此link，所以要手动操作:

cd /usr/local/include
ln -s ../opt/openssl/include/openssl .

可以解决编译过程中会出现的如下问题

[exec] -- Configuring incomplete, errors occurred!
[exec] See also /Users/user/github/hadoop/hadoop-tools/hadoop-pipes/target/native/CMakeCMake Error at /usr/local/Cellar/cmake/3.14.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
[exec]   Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec]   system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR)
[exec] Call Stack (most recent call first):
[exec]   /usr/local/Cellar/cmake/3.14.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
[exec]   /usr/local/Cellar/cmake/3.14.3/share/cmake/Modules/FindOpenSSL.cmake:413 (find_package_handle_stFiles/CMakeOutput.log.
[exec] andard_args)
[exec]   CMakeLists.txt:20 (find_package)

设置环境变量

export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HADOOP_HOME}/lib/native
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:${HADOOP_HOME}/lib/native

检查本地库

$ hadoop checknative -a

19/05/23 11:43:13 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
19/05/23 11:43:13 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/local/Cellar/hadoop/2.7.1/libexec/lib/native/libhadoop.dylib
zlib:    true /usr/lib/libz.1.dylib
snappy:  true /usr/local/lib/libsnappy.1.dylib
lz4:     true revision:99
bzip2:   false
openssl: true /usr/lib/libcrypto.35.dylib
19/05/23 11:43:13 INFO util.ExitUtil: Exiting with status 1