Windows 10编译Hadoop 2.6.0源码

最新推荐文章于 2024-09-05 12:49:15 发布

远太狼

最新推荐文章于 2024-09-05 12:49:15 发布

阅读量4.3k

点赞数

分类专栏： Hadoop 文章标签： hadoop 大数据 windows 源码

Hadoop 专栏收录该内容

0 篇文章 0 订阅

订阅专栏

转载自：https://www.zybuluo.com/ncepuwanghui/note/343755
原作者：Sam Wong 　　

向原文作者的辛勤劳动致敬！

版本控制

@Title          Windows 10编译Hadoop 2.6.0源码
@Version        v1.0
@Timestamp      2016-08-22 16:21
@Author         Sam Wong
@Mail           ncepuwanghui@foxmail.com

系统环境

@OS             Windows 10 Enterprise (64bit)
@JDK            JDK 1.7.0_67
@Hadoop         Hadoop 2.6.0

下载相关软件包

Building on Windows

Requirements:

Windows System JDK 1.6+ Maven 3.0 or later Findbugs 1.3.9 (if running
findbugs) ProtocolBuffer 2.5.0 CMake 2.6 or newer Windows SDK or
Visual Studio 2010 Professional Unix command-line tools from GnuWin32
or Cygwin: sh, mkdir, rm, cp, tar, gzip zlib headers (if building
native code bindings for zlib) Internet connection for first build (to
fetch all Maven and Hadoop dependencies)

If using Visual Studio, it must be Visual Studio 2010 Professional
(not 2012). Do not use Visual Studio Express. It does not support
compiling for 64-bit, which is problematic if running a 64-bit system.

注：除了下载protobuf源码外，还需要下载相应版本的编译过的用于Windows平台的protoc命令（protoc-2.5.0-win32.zip）,该命令用于将.proto文件转化为Java或C++源文件。

CMake

cmake-3.6.1-win64-x64.zip
下载地址：https://cmake.org/download/

Visual Studio 2010 Professional

Cygwin

setup-x86_64.exe
下载地址：https://cygwin.com/install.html

Zlib headers

zlib128.zip
下载地址：http://www.zlib.net/

安装配置所需环境

JDK

① 安装JDK
② 新建系统变量JAVA_HOME=C:\Progra~1\Java\jdk1.7.0_67

注：JAVA_HOME路径不能有空格，如果安装路径名中有空格，要使用旧式的8.3名字格式，即C:\Program Files\要写成C:\Progra~1\。

③ 编辑系统变量Path，新建%JAVA_HOME%\bin和%JAVA_HOME%\jre\bin

Maven

① 解压Maven到指定目录
② 新建系统变量M2_HOME=C:\WorkSpace\apache-maven-3.3.9
② 编辑系统变量Path，新建%M2_HOME%\bin

Findbugs

① 解压Findbugs到指定目录
② 编辑系统变量Path，新建C:\WorkSpace\findbugs-1.3.9\bin

ProtocolBuffer

① 解压ProtocolBuffer到指定目录
② 解压protoc-2.5.0-win32.zip，将protoc.exe复制到C:\WorkSpace\protobuf-2.5.0\src目录下
③ 安装ProtocolBuffer，打开CMD命令行

cd C:\WorkSpace\protobuf-2.5.0\java
mvn test 
mvn install
protoc --version

若没有protoc.exe或复制目录不正确，会提示如下错误：

[INFO] Executing tasks
    [mkdir] Created dir: C:\WorkSpace\protobuf-2.5.0\java\target\generated-sources
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:22 min
[INFO] Finished at: 2016-08-22T17:56:19+08:00
[INFO] Final Memory: 16M/439M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (generate-sources) on project protobuf-java: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "..\src\protoc": CreateProcess error=2, ????????? -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

④ 将protoc.exe所在路径C:\WorkSpace\protobuf-2.5.0\src，添加到系统变量Path

CMake

① 解压CMake到指定目录
② 编辑系统变量Path，新建C:\WorkSpace\cmake-3.6.1-win64-x64\bin

Cygwin

① 安装Cygwin64
② 编辑系统变量Path，新建C:\WorkSpace\cygwin64\bin

Zlib

① 解压Zlib到指定目录
② 新建系统变量ZLIB_HOME=C:\WorkSpace\zlib-1.2.8

编译Hadoop2.6.0

设置Platform

新建系统变量Platform=x64

编译Hadoop2.6.0

用右键以管理员身份运行Visual Studio x64 Win64 命令提示(2010)，开始菜单–>所有程序–>Microsoft Visual Studio 2010–>Visual Studio Tools–>Visual Studio x64 Win64 命令提示(2010)，执行如下命令：

cd C:\WorkSpace\hadoop-2.6.0-src
mvn package  -Pdist,native-win -DskipTests -Dtar

编译完成后，编译好的hadoo-2.6.0在目录C:\WorkSpace\hadoop-2.6.0-src\hadoop-dist\target\中。

注：本文采用Visual Studio 2015编译，开始–>所有应用–>Visual Studio 2015–>VS2015 x64
本机工具命令提示符

遇到的错误

① hadoop-common编译失败

[INFO] --- hadoop-maven-plugins:2.6.0:version-info (version-info) @ hadoop-common ---
[WARNING] [svn, info] failed: java.io.IOException: Cannot run program "svn": CreateProcess error=2, ?????????
[WARNING] [git, branch] failed with error code 128
[INFO] SCM: NONE
......
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.Cpp.Platform.targets(55,5): error MSB8020: The build tools for Visual Studio 2010 (Platform Toolset = 'v100') cannot be found. To build using the v100 build tools, please install Visual Studio 2010 build tools.  Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Retarget solution".[C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj]
已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj
”(默认目标)的操作 - 失败。
已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.vcxproj.metaproj”(默认目标)的操作 - 失败。
已完成生成项目“C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln”(默认
目标)的操作 - 失败。
生成失败。
......
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-comm
on: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]
org.ap
......
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

解决方法，Visual Studio版本问题，官方默认使用Visual Studio 2010 Professional，但本文采用Visual Studio 2015，因此对于生成失败的项目，需要用Visual Studio 2015重新打开，升级项目至Visual Studio 2015版本即可：

C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln
C:\WorkSpace\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native\native.sln

② hadoop-hdfs编译失败

main:
    [mkdir] Created dir: C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native
     [exec] -- The C compiler identification is unknown
     [exec] -- The CXX compiler identification is unknown
     [exec] CMake Error in CMakeLists.txt:
     [exec]   No CMAKE_C_COMPILER could be found.
     [exec] CMake Error in CMakeLists.txt:
     [exec]   No CMAKE_CXX_COMPILER could be found.
     [exec] -- Configuring incomplete, errors occurred!
     [exec] See also "C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeOutput.log".
     [exec] See also "C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log".
......
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-hdfs

问题原因是Visual Studio版本问题，解决方法：打开文件C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\pom.xml`，将

-G 'Visual Studio 10 Win64'

修改为：

-G 'Visual Studio 14 Win64'

若仍无法编译，可查看日志C:/WorkSpace/hadoop-2.6.0-src/hadoop-hdfs-project/hadoop-hdfs/target/native/CMakeFiles/CMakeError.log。若日志提示找不到Microsoft.Cpp.Default.props，可在C:\Program Files (x86)中查找Microsoft.Cpp.Default.props，例如最后找到所在路径C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140
用文本编辑器打开项目文件

C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\3.6.1\CompilerIdC\CompilerIdC.vcxproj
C:\WorkSpace\hadoop-2.6.0-src\hadoop-hdfs-project\hadoop-hdfs\target\native\CMakeFiles\3.6.1\CompilerIdCXX\CompilerIdCXX.vcxproj

查找Microsoft.Cpp.Default.props所在行，如下：

<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />

原因是没有系统变量VCTargetsPath，新建系统变量VCTargetsPath=C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140，然后重新编译：

mvn package  -Pdist,native-win -DskipTests -Dtar -rf :hadoop-hdfs

注：若还提示找不到，可能是环境变量未生效，关闭命令行窗口，重新打开执行编译命令，或者重启电脑试试。

参考链接

Build and Install Hadoop 2.x or newer on Windows
Windows 10下编译安装Hadoop2.6
在windows 7 64bit下编译Hadoop 2.6.0源码
 编译本地64位版本的hadoop-2.6.0
在Windows下编译Hadoop2.5
Protobuf 的安装使用总结

远太狼

关注

0
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
复制链接

分享到 QQ

分享到新浪微博

扫一扫

专栏目录