【Hadoop实战】Windows环境下编译Hadoop2（2.10.2-R0）

Windows环境下编译Hadoop2（2.10.2-R0）IDE

前提

根据Hadoop源码包解压之后编译帮助文件BUILDING.txt中关于windows的要求来准备环境

----------------------------------------------------------------------------------

Building on Windows

----------------------------------------------------------------------------------
Requirements:

* Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
  tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')

Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK 7.1 is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

The Windows SDK 8.1 is available to download at:

http://msdn.microsoft.com/en-us/windows/bg162891.aspx

Cygwin is neither required nor supported.

本机环境

• JDK1.8
• Windows 10 64位专业版
• maven 3.9.2
• git 2.41.0
• ProtocolBuffer 2.5.0 这个要求要满足不然有报错（Github地址）
• cmake 2.36.4
• Visual Studio 2022 Professional
• cygwin 安装包
• IDEA 2022.2.5

编译方式

使用VS2022的x86 Native Tools Command Prompt for VS 2022命令窗口进行编译，窗口位置C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Visual Studio 2022\Visual Studio Tools\VC

mvn clean package -Pdist -DskipTests

问题小结

安装Visual Studio之后，所有问题几乎全部解决。

问题1

hadoop-annotations: Compilation failure: Compilation failure:
[ERROR] /F:/open-source-code/hadoop/hadoop-release-2.10.2-RC0/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:[20,23] ▒▒▒▒▒com.sun.javadoc▒▒▒▒▒▒

确定好JDK的版本

问题2

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (compile-ms-winutils) on project hadoop-common: Command execution failed.: Cannot run program "msbuild" (in directory "F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-common"): CreateProcess error=2, ϵͳ▒Ҳ▒▒▒ָ▒▒▒▒▒ļ▒▒▒ -> [Help 1]

msbuild是微软的一个编译工具，这里提示找不到msbuild，因此可能是没有安装msbuild或没有配置到环境变量中。下面这样是不行的，还会报其他的错，正确的方式是需要下载Visual Studio，并将msbuild的路径（如，C:\Program Files (x86)\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin）加入到系统环境变量Path中。

# 因为这个目录下有msbuild.exe文件
C:\Windows\Microsoft.NET\Framework\v4.0.30319

问题3

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (compile-ms-winutils) on project hadoop-common: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

方式一

通过VS2022打开，并重定向F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln和F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-common\src\main\native\native.sln

方式二

通过注释的内容来跳过，在hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-common项目下的pom.xml中

          <!--<plugin>
             <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <executions>
              <execution>
                <id>compile-ms-winutils</id>
                <phase>compile</phase>
                <goals>
                  <goal>exec</goal>
                </goals>
                <configuration>
                  <executable>msbuild</executable>
                  <arguments>
                    <argument>${basedir}/src/main/winutils/winutils.sln</argument>
                    <argument>/nologo</argument>
                    <argument>/p:Configuration=Release</argument>
                    <argument>/p:OutDir=${project.build.directory}/bin/</argument>
                    <argument>/p:IntermediateOutputPath=${project.build.directory}/winutils/</argument>
                    <argument>/p:WsceConfigDir=${wsce.config.dir}</argument>
                    <argument>/p:WsceConfigFile=${wsce.config.file}</argument>
                  </arguments>
                </configuration>
              </execution>
              <execution>
                <id>compile-ms-native-dll</id>
                <phase>compile</phase>
                <goals>
                  <goal>exec</goal>
                </goals>
                <configuration>
                  <executable>msbuild</executable>
                  <arguments>
                    <argument>${basedir}/src/main/native/native.sln</argument>
                    <argument>/nologo</argument>
                    <argument>/p:Configuration=Release</argument>
                    <argument>/p:OutDir=${project.build.directory}/bin/</argument>
                    <argument>/p:CustomSnappyPrefix=${snappy.prefix}</argument>
                    <argument>/p:CustomSnappyLib=${snappy.lib}</argument>
                    <argument>/p:CustomSnappyInclude=${snappy.include}</argument>
                    <argument>/p:RequireSnappy=${require.snappy}</argument>
                    <argument>/p:CustomZstdPrefix=${zstd.prefix}</argument>
                    <argument>/p:CustomZstdLib=${zstd.lib}</argument>
                    <argument>/p:CustomZstdInclude=${zstd.include}</argument>
                    <argument>/p:RequireZstd=${require.zstd}</argument>
                    <argument>/p:CustomOpensslPrefix=${openssl.prefix}</argument>
                    <argument>/p:CustomOpensslLib=${openssl.lib}</argument>
                    <argument>/p:CustomOpensslInclude=${openssl.include}</argument>
                    <argument>/p:RequireOpenssl=${require.openssl}</argument>
                  </arguments>
                </configuration>
              </execution>
            </executions>
          </plugin> -->

问题4

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project hadoop-kms: An Ant BuildException has occured: java.net.ConnectException: Connection timed out: connect
[ERROR] around Ant part ...<get skipexisting="true" src="https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.75/bin/apache-tomcat-8.5.75.tar.gz" dest="downloads/apache-tomcat-8.5.75.tar.gz" verbose="true"/>... @ 5:183 in F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-kms\target\antrun\build-main.xml
[ERROR] -> [Help 1]

（有点挫，不知道为啥下不下来，单独从浏览器是可以下载的）

手动下载，放到对应的目录上，然后在注释的内容，在hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-kms项目下的pom.xml中

 <!-- <get
      src="${tomcat.download.url}"
      dest="downloads/apache-tomcat-${tomcat.version}.tar.gz"
      verbose="true" skipexisting="true"/> -->

Apache Hadoop HttpFS模块同样的问题，同样的处理。

问题5

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">... @ 8:166 in F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR] -> [Help 1]

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo does not exist.
[ERROR] around Ant part ...<copy todir="F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 14:127 in F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR] -> [Help 1]

应该还是跟native的环境要求有关，这里选择了跳过，在hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-kms项目下的pom.xml中，failonerror的值改为false，增加新建文件夹RelWithDebInfo的方法。

<configuration>
                  <target>
                    <mkdir dir="native/bin/RelWithDebInfo"/>
                    <condition property="generator" value="Visual Studio 10" else="Visual Studio 10 Win64">
                      <equals arg1="Win32" arg2="${env.PLATFORM}" />
                    </condition>
                    <mkdir dir="${project.build.directory}/native"/>
                    <exec executable="cmake" dir="${project.build.directory}/native"
                          failonerror="false">
                      <arg line="${basedir}/src/ -DGENERATED_JAVAH=${project.build.directory}/native/javah -DJVM_ARCH_DATA_MODEL=${sun.arch.data.model} -DREQUIRE_FUSE=${require.fuse} -G '${generator}'"/>
                    </exec>
                    <exec executable="msbuild" dir="${project.build.directory}/native"
                          failonerror="false">
                      <arg line="ALL_BUILD.vcxproj /nologo /p:Configuration=RelWithDebInfo /p:LinkIncremental=false"/>
                    </exec>
                    <!-- Copy for inclusion in distribution. -->
                    <copy todir="${project.build.directory}/bin">
                      <fileset dir="${project.build.directory}/native/bin/RelWithDebInfo"/>
                    </copy>
                  </target>
                </configuration>

问题6

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on project hadoop-hdfs-native-client: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

以下是互联网上的一种解法，但是在我未安装VS2022时并没有解决。然后是通过安装VS2022之后，使用x86 Native Tools Command Prompt for VS 2022命令窗口执行编译命令成功的。

下载安装Cygwin

配置cygwin64\bin到PATH

F:\software\cygwin64\bin

然后打开cygwin64的terminal窗口，执行命令

 mvn package -Pdist -DskipTests -Dtar -e -X -rf :hadoop-hdfs-native-client

导入IDEA

编译成功后使用IEDA打开源码即可，在打开的时候，会下载一些maven插件，可使用阿里云的maven仓库。