hadoop-3.0.1源码编译需要注意的事项

这次尝试了一下源码编译最新的hadoop3.0.1，发现了几个和原来不太一样的地方。记录下来：

1、需要的jdk不再是原来的1.7，直接jdk1.8就编译通过了；

2、以前安装需要安装编译依赖cmake,这次是需要cmake3，redhat7默认没有cmake3的package，需要先安装epel-release,然后安装cmake3,最后将cmake3做一个软链cmake；

yum install epel-release -y
yum install cmake3 -y
ln -s /usr/bin/cmake3 /usr/bin/cmake

3、编译hadoop amazon webservice时遇到缺少依赖包:DynamoDBLocal-1.11.86.jar。编译不通过，这时需要下载amazon提供的dynamodb_local_latest.tar.gz，解压之后，将其中的DynamoDBLocal.jar包拷贝到/path/maven/repo目录下的com/amazon/1.11.86/目录下，并命名为DynamoDBLocal-1.11.86.jar。接着编译就可以成功；

Failure to find com.amazonaws:DynamoDBLocal:jar

另外，网上有说更改hadoop-3.0.1-src目录下的pom.xm文件，增加dynamodb的仓库，我试了没有成功，不知道什么原因。

<repository>
       <id>dynamodb-local-oregon</id>
       <name>DynamoDB Local Release Repository</name>
       <url>https://s3-ap-southeast-1.amazonaws.com/dynamodb-local-singapore/release</url>
    </repository>

最终不得不下载dynamodb_local最新文件，解压出其中的DynamoDBLocal.jar。

4、其余的条件基本没什么变化。gcc-c++,openssl-devel,ncurses-devel和编译时需要的protobuf-2.5.0，再一个就是jdk,maven,编译时需要下载很多依赖包。

最后附上一个编译成功的打印信息：

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  0.606 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.187 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.588 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.350 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.093 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.144 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.612 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.159 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  3.331 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  1.520 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 36.657 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.024 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  2.565 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.030 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 10.304 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 33.907 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  1.186 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [  3.713 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  1.756 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.021 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.021 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  6.282 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 19.870 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.022 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  5.239 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  2.523 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [  5.820 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  1.822 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  2.954 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  2.634 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [  9.594 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.661 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  2.662 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  1.957 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  2.036 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  3.636 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  1.781 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  2.504 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.020 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.590 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.189 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.019 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.019 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  5.579 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.119 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [  9.071 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  7.961 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  2.258 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  3.789 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  2.843 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  2.078 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.134 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  3.074 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  2.565 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  3.236 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  2.518 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  2.035 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.245 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.160 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  2.598 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  1.777 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.260 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.078 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.092 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.321 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  4.232 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  2.446 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  5.362 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  2.752 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  1.984 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.827 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.193 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  4.739 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  2.395 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  6.701 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.023 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [01:20 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:02 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  0.592 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:31 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.106 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.072 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 16.725 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.018 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.629 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.014 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:51 min
[INFO] Finished at: 2018-04-05T16:25:41+08:00
[INFO] Final Memory: 173M/931M
[INFO] ------------------------------------------------------------------------

编译成功之后，编译结果在hadoop-3.0.1-src/hadoop-dist/target目录下，

我们可以查看一下编译之后，形成的lib/native/libhadoop.so.1.0.0。

[root@server target]# cd hadoop-3.0.1/lib/native/
[root@server native]# ls
examples     libhadooppipes.a  libhadoop.so.1.0.0  libhdfs.a   libhdfs.so.0.0.0  libnativetask.so
libhadoop.a  libhadoop.so      libhadooputils.a    libhdfs.so  libnativetask.a   libnativetask.so.1.0.0
[root@server native]# file libhadoop.so.1.0.0 
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=12cc31646dffc9651647fe12d8fe28b2dbc6280f, not stripped

本地依赖库有了之后，以后运行hadoop命令时就不会每次都提醒缺少本地库的警告了。