window10 下编译hadoop2.9.2

本文参考：https://blog.csdn.net/weixin_43827951/article/details/90311565
文中有一些地方不太一样已经改正，上面这篇博客博主还是挺用心的 -_-

Win10 下编译hadoop

一、解压源码

Hadoop2.9.2 源码下载：hadoop-2.9.2src.tar.gz源码下载 https://github.com/apache/hadoop/archive/release-2.9.2-RC0.zip
这里解压到了D:\cygwin64，并且重命名，D:\cygwin64\hadoop
参考源码里的 BUILDING.txt

二、工具

1. jdk-8u144-windows-x64	4. apache-maven-3.1.1-bin.zip	7. windows sdk 7.1
2. Git-2.21.0-64-bit.exe	5. zlib127-dll.zip
3. cmake-3.14.4-win64-x64	6. ProtocolBuffer 2.5.0	9. vs2010

三、工具安装配置

1. windows sdk 7.1
   下载地址:http://download.microsoft.com/download/F/1/0/F10113F5-B750-4969-A255-274341AC6BCE/GRMSDKX_EN_DVD.iso ,用UltraISO加载 或者 WinRAR解压;点击setup.exe安装；
   
2. vs2010
   先卸载比vs2010更高达版本;
   网盘中下载Visual Studio 2010 Ultimate.7z https://pan.baidu.com/s/1sldAQmD#list/path=%2F&parentPath=%2F ，解压,点击Install.exe(解压目录 就是安装目录)
   win sdk要先于vs2010安装；
3. jdk安装
   下载jdk-8u144-windows-x64.rar,然后安装;
   配置JAVA_HOME，添加%JAVA_HOME%\bin到PATH
   
   
4. Git-2.21.0-64-bit.exe
   下载:https://www.git-scm.com/download/win ，安装
   配置D:\home\git\cmd到PATH
   
5. cmake-3.14.4-win64-x64
   下载：https://cmake.org/download/ ,安装
6. Cygwin
   下载：http://www.cygwin.com/setup-x86_64.exe ,安装
   配置D:\cygwin64\bin到PATH
7. ===cygwin64这个软件不需要了，具体解决方案，看4编译的3小段
8. 
   
9. zlib127-dll.zip
   下载：https://excellmedia.dl.sourceforge.net/project/libpng/zlib/1.2.7/zlib127-dll.zip ,解压,配置ZLIB_HOME
   
10. ProtocolBuffer 2.5.0
    下载:https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0 ，下载protoc-2.5.0-win32.zip，解压后把 protoc.exe;放到 C:\Windows\System32。
11. apache-maven-3.1.1-bin.zip
    下载: https://mirrors.tuna.tsinghua.edu.cn/apache/maven/binaries/apache-maven-3.1.1-bin.zip ，解压到d:\home\maven3,配置M3_HOME,把%M3_HOME%\bin添加到PATH
    
    配置conf\setting.xml;

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">
     <localRepository>D:\mavenlib\repository</localRepository>
      <mirrors>
        <mirror>  
            <id>alimaven</id>  
            <name>aliyun maven</name>  
            <url>http://maven.aliyun.com/nexus/content/groups/public/</url>  
            <mirrorOf>central</mirrorOf>          
        </mirror>  
      </mirrors>
</settings>

 
 

 
 
1
2
3
4
5
6
7
8
9
10
11
12

四、编译

1. set Platform=x64
   

2. 添加msbuild.exe到PATH: 路径 c/Windows/Microsoft.NET/Framework/v4.0.30319/
   set PATH=%PATH%;c:\Windows\Microsoft.NET\Framework/v4.0.30319

3. 打开控制台;
   一般我们都是在idea Termina窗口操作，但是直接用idea默认Termina配置是不行的，因为idea通常会依赖高于VS2010版本的C++版本库（本人把系统高于2010版本的C++库删了，结果idea起不来了，！！！所以你懂得），所以我们不能使用idea的C++高版本库，当然也不能为了编译hadoop删了这些高版本的C++库，有好多软件需要呢！现在我们就要配置一下idea的Termina，=》1，先找到要配置的内容 ：window开始->所有程序->Microsoft Visual Studio 2010->Visual Studio Tools->Visual Studio x64兼容工具，找到这个快捷方式，查看属性，目标：%comspec% /k ""D:\software\Visual Studio 2010 Ultimate\VC\vcvarsall.bat"" x86_amd64 ，你们安装的就不是这个目录了，直接把%comspec% 替换成：C:\windows\system32\cmd.exe ，然后idea =》setting=》Tools=》Terminal=》shell path :C:\windows\system32\cmd.exe ""D:\software\Visual Studio 2010 Ultimate\VC\vcvarsall.bat"" x86_amd64

4. 进入到hadoop目录;

[hadoop]$ cd /hadoop/
[hadoop]$ mvn package -Pdist,native-win -DskipTests -Dtar

 
 

 
 
1
2

5. 编译OK
   
   编译后文件的路径: D:\cygwin64\hadoop\hadoop-dist\target
   发布的hadoop文件路径:D:\cygwin64\hadoop\hadoop-dist\target\hadoop-2.7.7
   

五、编译中遇到的问题总结

1. 缺失Unix command-line tools
   [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (pre-dist) on project hadoop-project-dist: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "sh" (in directory "E:\BigdData\idea\hadoop\hadoop-project-dist\target"): CreateProcess error=2, 系统找不到指定的文件。 [ERROR] around Ant part ...<exec failonerror="true" dir="E:\BigdData\idea\hadoop\hadoop-project-dist\target" executable="sh">... @ 41:111 in E:\BigdData\idea\hadoop\hadoop-project-dist\target\antrun\build-main.xml [ERROR] -> [Help 1]
   解决办法:安装cygwin,并把 D:\cygwin\bin 配置到 PATH
2. msbuild构造失败
   [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Cannot run program "msbuild" (in directory "D:\cygwin64\hadoop\hadoop-common-project\hadoop-common"): CreateProcess error=2, ϵͳ▒Ҳ▒▒▒ָ▒▒▒▒▒ļ▒▒▒ -> [Help 1] [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
   解决办法: 这个问题比较麻烦,有较高版本vs和msbuild,最后全部卸载,重新安装vs2010,添加msbuild到PATH下;
   set PATH=%PATH%;c:\Windows\Microsoft.NET\Framework/v4.0.30319