Mac 下编译 Hadoop-2.8

系统基本环境准备

由于下面列的东西都比较常规，具体的安装过程就不详细列举的，有问题可以自行百度

1. maven：Apache Maven 3.5.4

2. jdk：1.8.0_201

3. brew：Homebrew 2.2.6

编译环境准备

1.cmake：

安装命令 brew install cmake

2.openssl：

安装命令 brew install openssl

3.snappy：

安装命令 brew install snappy

4.zlib：

安装命令 brew install zlib

5.protobuf：

由于必须使用2.5.0版本需要自行下载安装https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0
下载后解压后进入protobuf目录
执行命令列表，进行编译
$ cd /Users/wx2514/tool/protobuf/protobuf-2.5.0
$ ./configure --prefix=/Users/wx2514/tool/protobuf/protobuf-2.5.0/
$ make
$ make install

配置环境变量修改 ~/.bash_profile
PATH=/Users/wx2514/tool/protobuf/protobuf-2.5.0/bin:$PATH
export PATH

验证protoc版本
$ protoc --version

编译hadoop

下载hadoop源码

https://github.com/apache/hadoop/tree/rel/release-2.8.0

开始编译

解压缩后进入hadoop源码路径

执行命令进行编译打包（这里打的不是最终可部署的压缩包）
$ mvn clean package -DskipTests

编译完成

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main 2.8.0 ........................... SUCCESS [  1.013 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.731 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.492 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  0.992 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.115 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.106 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.072 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.387 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  2.695 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  0.984 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 15.788 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  1.029 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  2.393 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.044 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [  8.803 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 13.615 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  0.309 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [  3.517 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  0.753 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  0.631 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.035 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.034 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  4.215 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [  5.345 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.032 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  1.794 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [  2.617 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  0.577 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  1.112 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [  4.702 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.852 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  0.915 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  0.662 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  0.840 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.023 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  0.513 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  0.424 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.045 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  0.888 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  0.305 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.131 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [  2.647 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  1.645 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  0.478 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  1.952 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  0.931 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  2.654 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  0.421 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  0.675 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  0.124 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  0.547 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  1.434 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  0.323 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  0.456 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  0.627 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  0.657 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  0.269 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  0.123 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  0.362 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.035 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  0.479 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  0.925 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  0.772 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  0.696 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.548 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  0.891 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  0.367 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  0.447 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.022 s]
[INFO] Apache Hadoop Distribution 2.8.0 ................... SUCCESS [  0.071 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:47 min
[INFO] Finished at: 2020-02-28T12:07:13+08:00
[INFO] ------------------------------------------------------------------------

编译部署包

备注：这个没在mac编译，没太大意义，编译的话也是去linux服务器上编译
mvn package -Pdist,native -DskipTests -Dtar

最终的包编译到了这个路径下面
hadoop-dist/target