vs2017编译haddoop3.1.0

最新推荐文章于 2022-09-01 21:48:21 发布

缥缈大师

最新推荐文章于 2022-09-01 21:48:21 发布

阅读量1k

点赞数

分类专栏：大数据文章标签： hadoop3.1.0 Windows编译 vs2017

本文链接：https://blog.csdn.net/cntpro/article/details/80679835

版权

大数据专栏收录该内容

1 篇文章 0 订阅

订阅专栏

我是在windows10 64位系统使用Visual Studio 2017编译hadoop 3.1.0源码

按照官方文档（BUILDING.txt），编译hadoop需要以下环境

1、 JDK 1.8

jdk-8u171-windows-x64.exe 下载地址

2、 Maven 3.0 or later

apache-maven-3.5.3-bin.zip下载地址

3、ProtocolBuffer 2.5.0

protoc-2.5.0-win32.zip下载地址

4、 CMake 3.1 or newer

cmake-3.11.3-win64-x64.zip下载地址

5、 Windows SDK 7.1 or Visual Studio 2010 Professional

6、 Windows SDK 8.1 (if building CPU rate control for the container executor)

我用的是vs2017，微软官方网站有下载。

7、zlib headers (if building native code bindings for zlib)

zlib-1.2.11.tar.gz 下载地址(可以不用)

8、 Internet connection for first build (to fetch all Maven and Hadoop dependencies)

联网是必须的

9、Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These tools must be present on your PATH.

可以用GnuWin32，MINGW64或者Git带的工具。

10、 Python ( for generation of docs using 'mvn site')

Python 3.6.5 下载地址（可以不用）

下载所需的工具，将他们路径加到你的环境变量里。

在命令行运行适用于 VS 2017 的 x64 本机工具命令提示，设置平台为set Platform=x64

一个简单的批处理，设置环境变量

@echo hadoop 编译环境设置
@echo 设置编译平台
@set Platform=x64

@echo 设置C编译环境
@call d:\tools\vs\2017\Community\VC\Auxiliary\Build\vcvars64.bat
@echo 加入maven路径
@set path=D:\tools\maven\bin;%path%
@echo 加入protc路径
@set path=D:\tools\protoc;%path%
@echo 加入bash路径
@set path=D:\tools\Git\bin;%path%
@echo 加入cmake路径
@set path=D:\tools\cmake\bin;%path%  由于默认是按照Visual Studio 2010环境配置的，我的是Visual Studio 2017，有几个文件需要修改一下

首先修改的是两个vs的工程文件

用vs2017打开hadoop-common-project\hadoop-common\src\main\native目录下native.sln文件，将工具和SDK定向到VS2017的版本如图

同理还有hadoop-common-project\hadoop-common\src\main\winutils目录下的winutils.sln，也要定向到vs2017的工具集

修改hadoop-hdfs-project\hadoop-hdfs-native-client目录下的pom.xml文件，将141行左右的

<condition property="generator" value="Visual Studio 10" else="Visual Studio 10 Win64">

改为

<condition property="generator" value="Visual Studio 10" else="Visual Studio 15 Win64">

cd进入源码目录，然后根据执行mvn package -Pdist,native-win -DskipTests -Dtar。等待编译完成，hadoop-dist\target\hadoop-3.1.0就有对应的hadoop可用目录

编译结果

[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop HDFS Native Client 3.1.0 ............. SUCCESS [ 46.213 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 16.726 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  5.358 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 24.532 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.113 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.094 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 37.818 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:27 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  9.039 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.080 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 25.877 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 34.372 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  3.654 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 10.063 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  7.387 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 58.543 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  1.698 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  9.020 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  3.970 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  3.358 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  0.079 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 11.089 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [  7.618 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [  0.083 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SUCCESS [  3.598 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [  4.049 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  5.197 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.089 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  3.055 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  2.027 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.173 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 53.781 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 24.747 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  3.953 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 15.146 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  7.131 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  9.522 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.985 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.075 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [  3.959 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [  1.262 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.092 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.073 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 18.167 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  2.072 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  6.425 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  1.995 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  8.095 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  5.299 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 12.191 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  9.260 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.863 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.992 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  8.485 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  4.835 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.299 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.983 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.067 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  5.312 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 23.122 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  2.702 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  7.845 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  3.313 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  4.009 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  6.494 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  6.009 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  3.185 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [  3.122 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.333 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.115 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [ 39.096 s]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [ 29.791 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  0.378 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [ 44.059 s]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.160 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.287 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [04:56 min]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.077 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.789 s]
[INFO] Apache Hadoop Cloud Storage Project 3.1.0 .......... SUCCESS [  0.081 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:37 min
[INFO] Finished at: 2018-06-14T09:04:32+08:00