centos7下编译hadoop

1.首先源码下载咯，请移步https://github.com/apache/hadoop

可以看到hadoop是java代码，是一个maven项目；然后hadoop有很多模块，目前还没看代码，不清楚具体细节。

2.其次依赖包安装

protobuf (这个要下载安装包，自行百度下载咯)
yum install gcc gcc-c++
yum install make, cmake
yum install openssl-devel
yum install ncurses-devel
maven (这个要3.3.0以上的版本，也自行下载安装咯)

其实一开始就把依赖包装好是对的，这样就不会在后面出现一个劲的报错，烦死人的。

友情提示：java代码报错不要想多了，要么换jdk，要么需要的有关c的编译器版本不对。

3.编译

mvn install eclipse:eclipse -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true

喝杯茶吧～
再喝杯茶吧～
喝第三杯茶吧～

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.562 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.920 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.208 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.699 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  1.002 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.026 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  4.072 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.485 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  5.545 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.080 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 36.739 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  4.427 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 18.488 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.549 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:29 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 54.037 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  6.020 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 27.315 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.663 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.536 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.567 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.401 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:45 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 22.585 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.636 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [01:05 min]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 23.961 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  3.104 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 17.317 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 10.346 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 10.898 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  3.953 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  3.589 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  3.008 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  4.292 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [02:26 min]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [01:48 min]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  3.897 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.581 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  2.975 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  2.305 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  1.422 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [  5.749 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  3.916 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  3.881 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  4.806 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  4.945 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  5.424 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  4.712 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.611 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 17.740 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [  3.333 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.511 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  1.742 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  9.354 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  3.185 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 34.305 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  2.587 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  3.749 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  4.795 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 13.467 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [06:25 min]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  2.087 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  2.832 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.250 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.719 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.180 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.094 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  4.994 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.141 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [05:42 min]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 23.902 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 31.438 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 17.201 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  4.773 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.988 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 16.148 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 51.428 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [  2.612 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 10.905 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.724 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [01:27 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:02 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  4.371 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:37 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  2.396 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.866 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 45.418 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.530 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  7.463 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.513 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 36:10 min
[INFO] Finished at: 2017-12-31T03:02:12-05:00
[INFO] Final Memory: 221M/792M
[INFO] ------------------------------------------------------------------------

12345，我们等的好辛苦！

尝试看看编译后的情况

[cheung@localhost bin]$ pwd
/home/cheung/workspace/hadoop/hadoop-dist/target/hadoop-3.1.0-SNAPSHOT/bin
[cheung@localhost bin]$ ./hadoop version
Hadoop 3.1.0-SNAPSHOT
Source code repository https://github.com/apache/hadoop.git -r 7fe6f83c8f0f67b1456c37d94b0de807e81a904a
Compiled by cheung on 2017-12-31T07:26Z
Compiled with protoc 2.5.0
From source with checksum 968f3bf47996a0b44b2435cc58855d2
This command was run using /home/cheung/workspace/hadoop/hadoop-dist/target/hadoop-3.1.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.1.0-SNAPSHOT.jar

4.Error & Solution

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-common: Compilation failure: Compilation failure: 
[ERROR] /home/cheung/workspace/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java:[40,21] cannot find symbol
[ERROR]   symbol:   class Cleaner
[ERROR]   location: package sun.misc
[ERROR] /home/cheung/workspace/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[333,17] cannot find symbol
[ERROR]   symbol:   class Cleaner
[ERROR]   location: package sun.misc
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

一开始用的是jdk1.9，将jdk版本降低到jdk1.8就好了

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-common: CMake failed with error code 1 -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

哎，cmake版本不对，蓝瘦香菇，cmake也最好弄到3.3吧

接下来就是一段冗长的代码阅读之旅.

Ready, GO!

参考资料
1.http://blog.csdn.net/xiexievv/article/details/47396725
2.http://blog.csdn.net/qianshangding0708/article/details/47679991