一、 背景
Hadoop官网提供的编译包如hadoop-2.7.4.tar.gz为Linux32位环境编译,其中native library在64位环境中运行会出现异常。因而在64位生产环境中使用的Hadoop是需要下载源代码并在64位系统中进行成功编译后的版本。
二、 环境
PC: MacBook Pro(Retina,13-inch,Mid 2014)
OS: macOS Sierra
三、 编译软件依赖
gcc |
GNU Autools Chain: autoconf, automake, libtool |
cmake |
snappy |
gzip |
bzip2 |
protobuf (使用protobuf@2.5) |
zlib |
openssl |
findbugs |
注意:
a) Hadoop编译依赖protobuf2.5.0版本,使用brew安装时不要安装最新版本,请使用如下地址protobuf@2.5
b) .bash_profile中配置环境变量
#Hadoop-common编译错误缺少 PATH="/usr/local/opt/protobuf@2.5/bin:${PATH}"
#mvn –Pdocs时缺少 FINDBUGS_HOME="/usr/local/opt/findbugs/libexec" export FINDBUGS_HOME PATH=".:${PATH}:${FINDBUGS_HOME}/bin" |
c) 升级OpenSSL
brew install openssl ln –s /usr/local/Cellar/openssl/1.0.2l/bin/openssl /usr/local/bin/openssl openssl version |
d) 详细依赖见src/ BUILDING.txt
安装命令
brew install gcc autoconf automake libtool cmake snappy gzip bzip2 protobuf@2.5 zlib openssl findbugs |
安装完毕后可以再次执行命令验证,结果如下
GavinGuo$ brew install gcc autoconf automake libtool cmake snappy gzip bzip2 protobuf@2.5 zlib openssl findbugs
Warning: gcc 7.2.0 is already installed Warning: autoconf 2.69 is already installed Warning: automake 1.15.1 is already installed Warning: libtool 2.4.6_1 is already installed Warning: cmake 3.9.1 is already installed Warning: snappy 1.1.7 is already installed Warning: gzip 1.8 is already installed Warning: bzip2 1.0.6_1 is already installed Warning: protobuf@2.5 2.5.0 is already installed Warning: zlib 1.2.11 is already installed Warning: openssl 1.0.2l is already installed |
安装软件依赖附录
GavinGuo$ brew install gcc autoconf automake libtool cmake snappy gzip bzip2 homebrew/versions/protobuf250 zlib openssl Warning: homebrew/versions was deprecated. This tap is now empty as all its formulae were migrated. ==> Tapping homebrew/versions Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-versions'... remote: Counting objects: 9, done. remote: Compressing objects: 100% (8/8), done. remote: Total 9 (delta 0), reused 5 (delta 0), pack-reused 0 Unpacking objects: 100% (9/9), done. Checking connectivity... done. Tapped 0 formulae (36 files, 31.0KB) Warning: Use protobuf@2.5 instead of deprecated homebrew/versions/protobuf250 Warning: cmake 3.9.1 is already installed Warning: openssl 1.0.2l is already installed ==> Installing dependencies for gcc:gmp, mpfr, libmpc, isl ==> Installing gcc dependency: gmp ==> Downloading https://homebrew.bintray.com/bottles/gmp-6.1.2.sierra.bottle.1.tar.gz ######################################################################## 100.0% ==> Pouring gmp-6.1.2.sierra.bottle.1.tar.gz �� /usr/local/Cellar/gmp/6.1.2: 18 files, 3.1MB ==> Installing gcc dependency: mpfr ==> Downloading https://homebrew.bintray.com/bottles/mpfr-3.1.5.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring mpfr-3.1.5.sierra.bottle.tar.gz �� /usr/local/Cellar/mpfr/3.1.5: 25 files, 3.6MB ==> Installing gcc dependency: libmpc ==> Downloading https://homebrew.bintray.com/bottles/libmpc-1.0.3_1.sierra.bottle.tar.g ######################################################################## 100.0% ==> Pouring libmpc-1.0.3_1.sierra.bottle.tar.gz �� /usr/local/Cellar/libmpc/1.0.3_1: 12 files, 345.0KB ==> Installing gcc dependency: isl ==> Downloading https://homebrew.bintray.com/bottles/isl-0.18.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring isl-0.18.sierra.bottle.tar.gz �� /usr/local/Cellar/isl/0.18: 80 files, 3.8MB ==> Installing gcc Warning: Building gcc from source: The bottle needs the Xcode CLT to be installed. ==> Downloading https://ftp.gnu.org/gnu/gcc/gcc-7.2.0/gcc-7.2.0.tar.xz ######################################################################## 100.0% ==> Downloading https://raw.githubusercontent.com/Homebrew/formula-patches/e9e0ee09389a ######################################################################## 100.0% ==> Patching ==> Applying 6.1.0-jit.patch patching file gcc/jit/Make-lang.in ==> ../configure --build=x86_64-apple-darwin16.7.0 --prefix=/usr/local/Cellar/gcc/7.2.0 ==> make ==> make install �� /usr/local/Cellar/gcc/7.2.0: 1,487 files, 284.7MB, built in 65 minutes 59 seconds ==> Downloading https://homebrew.bintray.com/bottles/autoconf-2.69.sierra.bottle.4.tar.gz ######################################################################## 100.0% ==> Pouring autoconf-2.69.sierra.bottle.4.tar.gz ==> Caveats Emacs Lisp files have been installed to: /usr/local/share/emacs/site-lisp/autoconf ==> Summary �� /usr/local/Cellar/autoconf/2.69: 70 files, 3.0MB ==> Downloading https://homebrew.bintray.com/bottles/automake-1.15.1.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring automake-1.15.1.sierra.bottle.tar.gz �� /usr/local/Cellar/automake/1.15.1: 131 files, 3.0MB ==> Downloading https://homebrew.bintray.com/bottles/libtool-2.4.6_1.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring libtool-2.4.6_1.sierra.bottle.tar.gz ==> Caveats In order to prevent conflicts with Apple's own libtool we have prepended a "g" so, you have instead: glibtool and glibtoolize. ==> Summary �� /usr/local/Cellar/libtool/2.4.6_1: 70 files, 3.7MB ==> Downloading https://homebrew.bintray.com/bottles/snappy-1.1.7.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring snappy-1.1.7.sierra.bottle.tar.gz �� /usr/local/Cellar/snappy/1.1.7: 18 files, 115.2KB ==> Downloading https://homebrew.bintray.com/bottles/gzip-1.8.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring gzip-1.8.sierra.bottle.tar.gz �� /usr/local/Cellar/gzip/1.8: 34 files, 326.5KB ==> Downloading https://homebrew.bintray.com/bottles/bzip2-1.0.6_1.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring bzip2-1.0.6_1.sierra.bottle.tar.gz ==> Caveats This formula is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble.
If you need to have this software first in your PATH run: echo 'export PATH="/usr/local/opt/bzip2/bin:$PATH"' >> ~/.bash_profile
For compilers to find this software you may need to set: LDFLAGS: -L/usr/local/opt/bzip2/lib CPPFLAGS: -I/usr/local/opt/bzip2/include
==> Summary �� /usr/local/Cellar/bzip2/1.0.6_1: 26 files, 413.6KB ==> Downloading https://homebrew.bintray.com/bottles/protobuf@2.5-2.5.0.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring protobuf@2.5-2.5.0.sierra.bottle.tar.gz ==> Caveats Editor support and examples have been installed to: /usr/local/opt/protobuf@2.5/share/doc/protobuf@2.5
This formula is keg-only, which means it was not symlinked into /usr/local, because this is an alternate version of another formula.
If you need to have this software first in your PATH run: echo 'export PATH="/usr/local/opt/protobuf@2.5/bin:$PATH"' >> ~/.bash_profile
For compilers to find this software you may need to set: LDFLAGS: -L/usr/local/opt/protobuf@2.5/lib CPPFLAGS: -I/usr/local/opt/protobuf@2.5/include
==> Summary �� /usr/local/Cellar/protobuf@2.5/2.5.0: 79 files, 5.4MB ==> Downloading https://homebrew.bintray.com/bottles/zlib-1.2.11.sierra.bottle.tar.gz ######################################################################## 100.0% ==> Pouring zlib-1.2.11.sierra.bottle.tar.gz ==> Caveats This formula is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble.
For compilers to find this software you may need to set: LDFLAGS: -L/usr/local/opt/zlib/lib CPPFLAGS: -I/usr/local/opt/zlib/include
==> Summary �� /usr/local/Cellar/zlib/1.2.11: 12 files, 374KB ==> Downloading https://downloads.sourceforge.net/project/findbugs/findbugs/3.0. ==> Downloading from https://jaist.dl.sourceforge.net/project/findbugs/findbugs/ ######################################################################## 100.0% ==> Caveats Some tools might refer findbugs by env variable. After installation, set $FINDBUGS_HOME in your profile: export FINDBUGS_HOME=/usr/local/opt/findbugs/libexec ==> Summary �� /usr/local/Cellar/findbugs/3.0.1: 158 files, 11.4MB, built in 9 seconds |
四、 编译Hadoop
1. 下载源代码并解压
https://archive.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4-src.tar.gz
将下载包放置到~/Desktop/
cd ~/Desktop/ tar -zxvf hadoop-2.7.4-src.tar.gz |
2. 使用mvn编译打包
cd ~/Desktop/hadoop-2.7.4-src mvn package -e -X -Pdist,native,docs -DskipTests -Dtar |
3. 安装成功日志
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 2.242 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.147 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.376 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.568 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.247 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.821 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.900 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 5.572 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 5.689 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.951 s] [INFO] Apache Hadoop Common ............................... SUCCESS [01:44 min] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 5.323 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 12.188 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.051 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [02:44 min] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 17.291 s] [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 4.338 s] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 3.498 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.045 s] [INFO] hadoop-yarn ........................................ SUCCESS [ 0.047 s] [INFO] hadoop-yarn-api .................................... SUCCESS [01:46 min] [INFO] hadoop-yarn-common ................................. SUCCESS [ 22.875 s] [INFO] hadoop-yarn-server ................................. SUCCESS [ 0.047 s] [INFO] hadoop-yarn-server-common .......................... SUCCESS [ 9.791 s] [INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 12.379 s] [INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 2.850 s] [INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 7.190 s] [INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 15.532 s] [INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 4.786 s] [INFO] hadoop-yarn-client ................................. SUCCESS [ 5.563 s] [INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 3.171 s] [INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.052 s] [INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 2.342 s] [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 1.579 s] [INFO] hadoop-yarn-site ................................... SUCCESS [ 0.039 s] [INFO] hadoop-yarn-registry ............................... SUCCESS [ 3.933 s] [INFO] hadoop-yarn-project ................................ SUCCESS [ 4.126 s] [INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.310 s] [INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 18.118 s] [INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 13.968 s] [INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 3.386 s] [INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 7.465 s] [INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 5.136 s] [INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 3.893 s] [INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 2.286 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 5.870 s] [INFO] hadoop-mapreduce ................................... SUCCESS [ 3.015 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 4.036 s] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 7.211 s] [INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.578 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 4.256 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 3.790 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 3.018 s] [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 3.237 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.757 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 7.702 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 4.220 s] [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 4.398 s] [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 3.822 s] [INFO] Apache Hadoop Client ............................... SUCCESS [ 9.447 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.508 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 6.128 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 8.155 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.044 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [01:27 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:50 min [INFO] Finished at: 2017-08-26T21:00:21+08:00 [INFO] Final Memory: 116M/706M [INFO] ------------------------------------------------------------------------
|
4. 获取编译后工程
编译成功的工程在如下路径
/Users/GavinGuo/Desktop/hadoop-2.7.4-src/hadoop-dist/target/hadoop-2.7.4 |
可直接将工程拷贝安装成为64位正式Hadoop工程或将其打包为tar文件备用
cd ~/Desktop/hadoop-2.7.4-src/hadoop-dist/target/ tar -czvf ~/Desktop/hadoop-2.7.4-x86_64.tar.gz hadoop-2.7.4 |
5. 验证
经验证Mac编译的native包生成了.dylib的库文件,而Linux使用的.so库文件,故Mac编译的Hadoop不支持在Linux上使用。Linux使用的64位Hadoop需要在Linux上编译生成。Linux编译Hadoop请参照《CentOS7.3编译Hadoop2.7.4》