手把手教你编译hadoop源码

最新推荐文章于 2023-10-06 21:09:36 发布

oyezitan

最新推荐文章于 2023-10-06 21:09:36 发布

阅读量699

点赞数

分类专栏：大数据/云计算

本文链接：https://blog.csdn.net/goodmentc/article/details/80960154

版权

大数据/云计算专栏收录该内容

12 篇文章 0 订阅

订阅专栏

一、版本选择

之前编译过最新版本3.0.3，不是很顺利。
这次选择2.7.6。

下载源码：
http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.6/hadoop-2.7.6-src.tar.gz
解压到：

/home/tancan/source/

二、编译环境准备

2.1 说明

Hadoop源码里有个BUILDING.txt文件，说明了编译需要的环境：

Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

 1. Unix System
 2. JDK 1.7+
 3. Maven 3.0 or later
 4. Findbugs 1.3.9 (if running findbugs)
 5. ProtocolBuffer 2.5.0
 6. CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
 7. Zlib devel (if compiling native code)
 8. openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
 9. Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
 10. Internet connection for first build (to fetch all Maven and Hadoop dependencies)

----------------------------------------------------------------------------------

2.2 安装需要的软件

1.jdk1.8
2.Maven

sudo apt-get install maven

检查安装结果mvn -v：

tancan@master:~/source$ mvn -v
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_111, vendor: Oracle Corporation
Java home: /usr/soft/jdk1.8.0_111/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.13.0-45-generic", arch: "i386", family: "unix"
tancan@master:~/source$

3.安装protobuf

tancan@master:~/source$ protoc --version
libprotoc 2.5.0

4.安装openssl
5.安装cmake
6.配置ssh免密钥登录
https://blog.csdn.net/goodmentc/article/details/80964193

开始编译

在hadoop源码目录下，执行编译命令：

mvn package -Pdist,native -DskipTests -Dtar
Or:
mvn package -Pdist,native,docs,src -DskipTests -Dtar

第二个命令的编译时间要久一些。
这里使用第一个命令。

tancan@master:~/source/hadoop-2.7.6-src$ mvn package -Pdist,native -DskipTests -Dtar

编译成功产生的日志：

[INFO] 
[INFO] -------------------< org.apache.hadoop:hadoop-dist >--------------------
[INFO] Building Apache Hadoop Distribution 2.7.6                        [65/65]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-dist ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /home/tancan/source/hadoop-2.7.6-src/hadoop-dist/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-dist ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-dist ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/tancan/source/hadoop-2.7.6-src/hadoop-dist/src/main/resources
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-dist ---
[INFO] No sources to compile
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main 2.7.6 ........................... SUCCESS [  0.674 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.127 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.646 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  0.236 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.112 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.087 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  1.082 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.474 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  1.829 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  0.199 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [07:33 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  0.974 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 49.277 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.069 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 28.714 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [01:14 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [02:27 min]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  0.785 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.047 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.043 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [  3.597 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [03:45 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.055 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  2.489 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 29.800 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  0.822 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  1.309 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [  2.537 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 25.263 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  5.364 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  0.671 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.039 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  0.439 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  0.246 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.028 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  0.781 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  0.268 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.100 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 58.341 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [  3.726 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  1.448 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  5.380 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 18.019 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 38.538 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  0.349 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 30.712 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  0.177 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 20.430 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  0.370 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  0.232 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  0.495 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  0.408 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  0.185 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  0.137 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  0.228 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  7.706 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  0.671 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [07:06 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 57.585 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  1.254 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.679 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  1.134 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  0.771 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.106 s]
[INFO] Apache Hadoop Distribution 2.7.6 ................... SUCCESS [  0.089 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29:01 min
[INFO] Finished at: 2018-07-09T07:53:06-07:00
[INFO] ------------------------------------------------------------------------
tancan@master:~/source/hadoop-2.7.6-src$

这里写图片描述

大功告成！
到这个路径寻找编译成果：

/home/tancan/source/hadoop-2.7.6-src/hadoop-dist/target

编译生成的Hadoop安装包：
编译生产的库文件：

编译过程遇到的问题

1.maven安装方式导致报错

[INFO] Using 'UTF-8' encoding to read mojo metadata.
[INFO] Mojo extractor with id: java-javadoc found 0 mojo descriptors.
[INFO] Mojo extractor with id: java-annotations found 2 mojo descriptors.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [ 35.781 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.999 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 17.957 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.275 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.100 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.941 s]
[INFO] Apache Hadoop Maven Plugins ........................ FAILURE [  1.994 s]
[INFO] Apache Hadoop MiniKDC .............................. SKIPPED
[INFO] Apache Hadoop Auth ................................. SKIPPED
[INFO] Apache Hadoop Auth Examples ........................ SKIPPED
[INFO] Apache Hadoop Common ............................... SKIPPED
[INFO] Apache Hadoop NFS .................................. SKIPPED
[INFO] Apache Hadoop KMS .................................. SKIPPED
[INFO] Apache Hadoop Common Project ....................... SKIPPED
[INFO] Apache Hadoop HDFS ................................. SKIPPED
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] hadoop-yarn ........................................ SKIPPED
[INFO] hadoop-yarn-api .................................... SKIPPED
[INFO] hadoop-yarn-common ................................. SKIPPED
[INFO] hadoop-yarn-server ................................. SKIPPED
[INFO] hadoop-yarn-server-common .......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED
[INFO] hadoop-yarn-server-tests ........................... SKIPPED
[INFO] hadoop-yarn-client ................................. SKIPPED
[INFO] hadoop-yarn-server-sharedcachemanager .............. SKIPPED
[INFO] hadoop-yarn-applications ........................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
[INFO] hadoop-yarn-site ................................... SKIPPED
[INFO] hadoop-yarn-registry ............................... SKIPPED
[INFO] hadoop-yarn-project ................................ SKIPPED
[INFO] hadoop-mapreduce-client ............................ SKIPPED
[INFO] hadoop-mapreduce-client-core ....................... SKIPPED
[INFO] hadoop-mapreduce-client-common ..................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED
[INFO] hadoop-mapreduce-client-app ........................ SKIPPED
[INFO] hadoop-mapreduce-client-hs ......................... SKIPPED
[INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] hadoop-mapreduce ................................... SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Ant Tasks ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:06 min
[INFO] Finished at: 2018-07-08T01:11:00-07:00
[INFO] Final Memory: 58M/393M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.4:descriptor (default-descriptor) on project hadoop-maven-plugins: The API of the mojo scanner is not compatible with this plugin version. Please check the plugin dependencies configured in the POM and ensure the versions match. /usr/soft/jdk1.8.0_111/jre/lib/i386/libawt_xawt.so: libXext.so.6: cannot open shared object file: No such file or directory -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-maven-plugins
tancan@master:~/source/hadoop-2.7.6-src$

错误原因：
使用sudo apt-get install maven安装了maven。
先卸载掉maven，再手动安装maven。
maven下载地址：

http://maven.apache.org/download.cgi

我下载的是：apache-maven-3.5.4-bin.tar.gz

安装方法：

解压安装包到指定路径；
设置环境变量：

tancan@master:~/source/hadoop-2.7.6-src$ cat /etc/environment 
JAVA_HOME=/usr/soft/jdk1.8.0_171

HADOOP_INSTALL=/usr/soft/hadoop-3.0.3


PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/soft/jdk1.8.0_171/bin:/usr/soft/hadoop-3.0.3/bin:/usr/soft/hadoop-3.0.3/sbin:/usr/soft/apache-maven-3.5.4/bin"

让环境变量生效：
tancan@master:~/source/hadoop-2.7.6-src$ source /etc/environment
验证安装是否成功：

tancan@master:~/source/hadoop-2.7.6-src$ mvn -v
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T11:33:14-07:00)
Maven home: /usr/soft/apache-maven-3.5.4
Java version: 1.8.0_171, vendor: Oracle Corporation, runtime: /usr/soft/jdk1.8.0_171/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.13.0-45-generic", arch: "amd64", family: "unix"
tancan@master:~/source/hadoop-2.7.6-src$

2.缺少maven文件报错：

[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main 2.7.6 ........................... SUCCESS [  0.254 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.945 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.759 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  0.152 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.073 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.089 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  0.862 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [37:41 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:40 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  0.316 s]
[INFO] Apache Hadoop Common ............................... FAILURE [  0.008 s]
[INFO] Apache Hadoop NFS .................................. SKIPPED
[INFO] Apache Hadoop KMS .................................. SKIPPED
[INFO] Apache Hadoop Common Project ....................... SKIPPED
[INFO] Apache Hadoop HDFS ................................. SKIPPED
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] hadoop-yarn ........................................ SKIPPED
[INFO] hadoop-yarn-api .................................... SKIPPED
[INFO] hadoop-yarn-common ................................. SKIPPED
[INFO] hadoop-yarn-server ................................. SKIPPED
[INFO] hadoop-yarn-server-common .......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED
[INFO] hadoop-yarn-server-tests ........................... SKIPPED
[INFO] hadoop-yarn-client ................................. SKIPPED
[INFO] hadoop-yarn-server-sharedcachemanager .............. SKIPPED
[INFO] hadoop-yarn-applications ........................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
[INFO] hadoop-yarn-site ................................... SKIPPED
[INFO] hadoop-yarn-registry ............................... SKIPPED
[INFO] hadoop-yarn-project ................................ SKIPPED
[INFO] hadoop-mapreduce-client ............................ SKIPPED
[INFO] hadoop-mapreduce-client-core ....................... SKIPPED
[INFO] hadoop-mapreduce-client-common ..................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED
[INFO] hadoop-mapreduce-client-app ........................ SKIPPED
[INFO] hadoop-mapreduce-client-hs ......................... SKIPPED
[INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] hadoop-mapreduce ................................... SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Ant Tasks ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution 2.7.6 ................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41:32 min
[INFO] Finished at: 2018-07-08T08:16:31-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to parse plugin descriptor for org.apache.hadoop:hadoop-maven-plugins:2.7.6 (/home/tancan/source/hadoop-2.7.6-src/hadoop-maven-plugins/target/classes): No plugin descriptor found at META-INF/maven/plugin.xml -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginDescriptorParsingException
tancan@master:~/source/hadoop-2.7.6-src$

错误原因分析：
缺少文件：No plugin descriptor found at META-INF/maven/plugin.xml

文件下载地址:
https://download.csdn.net/download/goodmentc/10531487

下载之后解压，放入此目录：
这里写图片描述

打开plugin.xml修改hadoop的版本号，如果你也是编译2.7.6版本，就不用修改了。
这里写图片描述

总结

前后编译了好几次，都没有成功，今天终于成功了，编译心得：

配套软件需要正确安装，尤其是maven；
补充缺少文件；
遇到错误谷歌。

下次重新编译一下最新版代码3.0.3。

顺便编译了一下3.0.3版本源码：
编译流程同上。
第一次编译报错：

INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main 3.0.3 ........................... SUCCESS [02:51 min]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 16.352 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 24.382 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 28.144 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.257 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 26.664 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 11.532 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [01:06 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:46 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  9.111 s]
[INFO] Apache Hadoop Common ............................... FAILURE [ 34.668 s]
[INFO] Apache Hadoop NFS .................................. SKIPPED
[INFO] Apache Hadoop KMS .................................. SKIPPED
[INFO] Apache Hadoop Common Project ....................... SKIPPED
[INFO] Apache Hadoop HDFS Client .......................... SKIPPED
[INFO] Apache Hadoop HDFS ................................. SKIPPED
[INFO] Apache Hadoop HDFS Native Client ................... SKIPPED
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS-RBF ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] Apache Hadoop YARN ................................. SKIPPED
[INFO] Apache Hadoop YARN API ............................. SKIPPED
[INFO] Apache Hadoop YARN Common .......................... SKIPPED
[INFO] Apache Hadoop YARN Server .......................... SKIPPED
[INFO] Apache Hadoop YARN Server Common ................... SKIPPED
[INFO] Apache Hadoop YARN Registry ........................ SKIPPED
[INFO] Apache Hadoop YARN NodeManager ..................... SKIPPED
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project 3.0.3 .......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:20 min
[INFO] Finished at: 2018-07-09T09:02:57-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.3:cmake-compile (cmake-compile) on project hadoop-common: make failed with error code 2 -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

失败原因：
估计是因为之前编译过2.7.6版本，有些文件冲突。

使用如下编译命令成功编译：

mvn clean package -Pdist,native -DskipTests -Dtar -e

这里写图片描述

oyezitan

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
打赏
0
评论
手把手教你编译hadoop源码

一、版本选择之前编译过最新版本3.0.3，不是很顺利。这次选择2.7.6。下载源码： http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.6/hadoop-2.7.6-src.tar.gz 二、编译环境准备2.1 说明Hadoop源码里有个BUILDING.txt文件，说明了编译需要的环...
复制链接

扫一扫