MacOS10编译Hadoop2.7.4

一、  背景

Hadoop官网提供的编译包如hadoop-2.7.4.tar.gz为Linux32位环境编译，其中native library在64位环境中运行会出现异常。因而在64位生产环境中使用的Hadoop是需要下载源代码并在64位系统中进行成功编译后的版本。

 

二、  环境

PC: MacBook Pro(Retina,13-inch,Mid 2014)

OS: macOS Sierra

 

三、  编译软件依赖

gcc

GNU Autools Chain: autoconf, automake, libtool

cmake

snappy

gzip

bzip2

protobuf (使用protobuf@2.5)

zlib

openssl

findbugs

 

注意:

a)  Hadoop编译依赖protobuf2.5.0版本，使用brew安装时不要安装最新版本，请使用如下地址protobuf@2.5

b)  .bash_profile中配置环境变量

#Hadoop-common编译错误缺少 PATH="/usr/local/opt/protobuf@2.5/bin:${PATH}"   #mvn –Pdocs时缺少 FINDBUGS_HOME="/usr/local/opt/findbugs/libexec" export FINDBUGS_HOME PATH=".:${PATH}:${FINDBUGS_HOME}/bin"

c)  升级OpenSSL

brew install openssl ln –s /usr/local/Cellar/openssl/1.0.2l/bin/openssl /usr/local/bin/openssl openssl version

 

d)  详细依赖见src/ BUILDING.txt

  

   安装命令

brew install gcc autoconf automake libtool cmake snappy gzip bzip2 protobuf@2.5 zlib openssl findbugs

 

   安装完毕后可以再次执行命令验证，结果如下

GavinGuo$ brew install gcc autoconf automake libtool cmake snappy gzip bzip2 protobuf@2.5 zlib openssl findbugs   Warning: gcc 7.2.0 is already installed Warning: autoconf 2.69 is already installed Warning: automake 1.15.1 is already installed Warning: libtool 2.4.6_1 is already installed Warning: cmake 3.9.1 is already installed Warning: snappy 1.1.7 is already installed Warning: gzip 1.8 is already installed Warning: bzip2 1.0.6_1 is already installed Warning: protobuf@2.5 2.5.0 is already installed Warning: zlib 1.2.11 is already installed Warning: openssl 1.0.2l is already installed

 

安装软件依赖附录

GavinGuo$ brew install gcc autoconf automake libtool cmake snappy gzip bzip2 homebrew/versions/protobuf250 zlib openssl Warning: homebrew/versions was deprecated. This tap is now empty as all its formulae were migrated. ==> Tapping homebrew/versions Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-versions'... remote: Counting objects: 9, done. remote: Compressing objects: 100% (8/8), done. remote: Total 9 (delta 0), reused 5 (delta 0), pack-reused 0 Unpacking objects: 100% (9/9), done. Checking connectivity... done. Tapped 0 formulae (36 files, 31.0KB) Warning: Use protobuf@2.5 instead of deprecated homebrew/versions/protobuf250 Warning: cmake 3.9.1 is already installed Warning: openssl 1.0.2l is already installed ==> Installing dependencies for gcc:gmp, mpfr, libmpc, isl ==> Installing gcc dependency: gmp ==> Downloading https://homebrew.bintray.com/bottles/gmp-6.1.2.sierra.bottle.1.tar.gz ######################################################################## 100.0% ==> Pouring gmp-6.1.2.sierra.bottle.1.tar.gz ��  /usr/local/Cellar/gmp/6.1.2: 18 files, 3.1MB ==> Installing gcc dependency: mpfr ==> Downloading https://homebrew.bintray.com/bottles/mpfr-3.1.5.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring mpfr-3.1.5.sierra.bottle.tar.gz ��  /usr/local/Cellar/mpfr/3.1.5: 25 files, 3.6MB ==> Installing gcc dependency: libmpc ==> Downloading https://homebrew.bintray.com/bottles/libmpc-1.0.3_1.sierra.bottle.tar.g ######################################################################## 100.0% ==> Pouring libmpc-1.0.3_1.sierra.bottle.tar.gz ��  /usr/local/Cellar/libmpc/1.0.3_1: 12 files, 345.0KB ==> Installing gcc dependency: isl ==> Downloading https://homebrew.bintray.com/bottles/isl-0.18.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring isl-0.18.sierra.bottle.tar.gz ��  /usr/local/Cellar/isl/0.18: 80 files, 3.8MB ==> Installing gcc Warning: Building gcc from source:   The bottle needs the Xcode CLT to be installed. ==> Downloading https://ftp.gnu.org/gnu/gcc/gcc-7.2.0/gcc-7.2.0.tar.xz ######################################################################## 100.0% ==> Downloading https://raw.githubusercontent.com/Homebrew/formula-patches/e9e0ee09389a ######################################################################## 100.0% ==> Patching ==> Applying 6.1.0-jit.patch patching file gcc/jit/Make-lang.in ==> ../configure --build=x86_64-apple-darwin16.7.0 --prefix=/usr/local/Cellar/gcc/7.2.0 ==> make ==> make install ��  /usr/local/Cellar/gcc/7.2.0: 1,487 files, 284.7MB, built in 65 minutes 59 seconds ==> Downloading https://homebrew.bintray.com/bottles/autoconf-2.69.sierra.bottle.4.tar.gz ######################################################################## 100.0% ==> Pouring autoconf-2.69.sierra.bottle.4.tar.gz ==> Caveats Emacs Lisp files have been installed to:   /usr/local/share/emacs/site-lisp/autoconf ==> Summary ��  /usr/local/Cellar/autoconf/2.69: 70 files, 3.0MB ==> Downloading https://homebrew.bintray.com/bottles/automake-1.15.1.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring automake-1.15.1.sierra.bottle.tar.gz ��  /usr/local/Cellar/automake/1.15.1: 131 files, 3.0MB ==> Downloading https://homebrew.bintray.com/bottles/libtool-2.4.6_1.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring libtool-2.4.6_1.sierra.bottle.tar.gz ==> Caveats In order to prevent conflicts with Apple's own libtool we have prepended a "g" so, you have instead: glibtool and glibtoolize. ==> Summary ��  /usr/local/Cellar/libtool/2.4.6_1: 70 files, 3.7MB ==> Downloading https://homebrew.bintray.com/bottles/snappy-1.1.7.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring snappy-1.1.7.sierra.bottle.tar.gz ��  /usr/local/Cellar/snappy/1.1.7: 18 files, 115.2KB ==> Downloading https://homebrew.bintray.com/bottles/gzip-1.8.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring gzip-1.8.sierra.bottle.tar.gz ��  /usr/local/Cellar/gzip/1.8: 34 files, 326.5KB ==> Downloading https://homebrew.bintray.com/bottles/bzip2-1.0.6_1.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring bzip2-1.0.6_1.sierra.bottle.tar.gz ==> Caveats This formula is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble.   If you need to have this software first in your PATH run:   echo 'export PATH="/usr/local/opt/bzip2/bin:$PATH"' >> ~/.bash_profile   For compilers to find this software you may need to set:     LDFLAGS:  -L/usr/local/opt/bzip2/lib     CPPFLAGS: -I/usr/local/opt/bzip2/include   ==> Summary ��  /usr/local/Cellar/bzip2/1.0.6_1: 26 files, 413.6KB ==> Downloading https://homebrew.bintray.com/bottles/protobuf@2.5-2.5.0.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring protobuf@2.5-2.5.0.sierra.bottle.tar.gz ==> Caveats Editor support and examples have been installed to:   /usr/local/opt/protobuf@2.5/share/doc/protobuf@2.5   This formula is keg-only, which means it was not symlinked into /usr/local, because this is an alternate version of another formula.   If you need to have this software first in your PATH run:   echo 'export PATH="/usr/local/opt/protobuf@2.5/bin:$PATH"' >> ~/.bash_profile   For compilers to find this software you may need to set:     LDFLAGS:  -L/usr/local/opt/protobuf@2.5/lib     CPPFLAGS: -I/usr/local/opt/protobuf@2.5/include   ==> Summary ��  /usr/local/Cellar/protobuf@2.5/2.5.0: 79 files, 5.4MB ==> Downloading https://homebrew.bintray.com/bottles/zlib-1.2.11.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring zlib-1.2.11.sierra.bottle.tar.gz ==> Caveats This formula is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble.   For compilers to find this software you may need to set:     LDFLAGS:  -L/usr/local/opt/zlib/lib     CPPFLAGS: -I/usr/local/opt/zlib/include   ==> Summary ��  /usr/local/Cellar/zlib/1.2.11: 12 files, 374KB ==> Downloading https://downloads.sourceforge.net/project/findbugs/findbugs/3.0. ==> Downloading from https://jaist.dl.sourceforge.net/project/findbugs/findbugs/ ######################################################################## 100.0% ==> Caveats Some tools might refer findbugs by env variable. After installation, set $FINDBUGS_HOME in your profile:   export FINDBUGS_HOME=/usr/local/opt/findbugs/libexec ==> Summary ��  /usr/local/Cellar/findbugs/3.0.1: 158 files, 11.4MB, built in 9 seconds

 

四、  编译Hadoop

1. 下载源代码并解压

https://archive.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4-src.tar.gz

将下载包放置到~/Desktop/

cd ~/Desktop/ tar -zxvf hadoop-2.7.4-src.tar.gz

 

2. 使用mvn编译打包

cd ~/Desktop/hadoop-2.7.4-src mvn package -e -X -Pdist,native,docs -DskipTests -Dtar

 

3. 安装成功日志

[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [  2.242 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.147 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.376 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.568 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.247 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.821 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.900 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  5.572 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [  5.689 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.951 s] [INFO] Apache Hadoop Common ............................... SUCCESS [01:44 min] [INFO] Apache Hadoop NFS .................................. SUCCESS [  5.323 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 12.188 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.051 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [02:44 min] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 17.291 s] [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  4.338 s] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.498 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.045 s] [INFO] hadoop-yarn ........................................ SUCCESS [  0.047 s] [INFO] hadoop-yarn-api .................................... SUCCESS [01:46 min] [INFO] hadoop-yarn-common ................................. SUCCESS [ 22.875 s] [INFO] hadoop-yarn-server ................................. SUCCESS [  0.047 s] [INFO] hadoop-yarn-server-common .......................... SUCCESS [  9.791 s] [INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 12.379 s] [INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.850 s] [INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  7.190 s] [INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 15.532 s] [INFO] hadoop-yarn-server-tests ........................... SUCCESS [  4.786 s] [INFO] hadoop-yarn-client ................................. SUCCESS [  5.563 s] [INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  3.171 s] [INFO] hadoop-yarn-applications ........................... SUCCESS [  0.052 s] [INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.342 s] [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.579 s] [INFO] hadoop-yarn-site ................................... SUCCESS [  0.039 s] [INFO] hadoop-yarn-registry ............................... SUCCESS [  3.933 s] [INFO] hadoop-yarn-project ................................ SUCCESS [  4.126 s] [INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.310 s] [INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 18.118 s] [INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 13.968 s] [INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  3.386 s] [INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  7.465 s] [INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  5.136 s] [INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  3.893 s] [INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  2.286 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  5.870 s] [INFO] hadoop-mapreduce ................................... SUCCESS [  3.015 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  4.036 s] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  7.211 s] [INFO] Apache Hadoop Archives ............................. SUCCESS [  1.578 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [  4.256 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.790 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [  3.018 s] [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  3.237 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [  2.757 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [  7.702 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [  4.220 s] [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  4.398 s] [INFO] Apache Hadoop Azure support ........................ SUCCESS [  3.822 s] [INFO] Apache Hadoop Client ............................... SUCCESS [  9.447 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.508 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  6.128 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  8.155 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [  0.044 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [01:27 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:50 min [INFO] Finished at: 2017-08-26T21:00:21+08:00 [INFO] Final Memory: 116M/706M [INFO] ------------------------------------------------------------------------

 

4. 获取编译后工程

编译成功的工程在如下路径

/Users/GavinGuo/Desktop/hadoop-2.7.4-src/hadoop-dist/target/hadoop-2.7.4

 

可直接将工程拷贝安装成为64位正式Hadoop工程或将其打包为tar文件备用

cd ~/Desktop/hadoop-2.7.4-src/hadoop-dist/target/ tar -czvf ~/Desktop/hadoop-2.7.4-x86_64.tar.gz hadoop-2.7.4

 

5. 验证

经验证Mac编译的native包生成了.dylib的库文件，而Linux使用的.so库文件，故Mac编译的Hadoop不支持在Linux上使用。Linux使用的64位Hadoop需要在Linux上编译生成。Linux编译Hadoop请参照《CentOS7.3编译Hadoop2.7.4》