CentOS7环境下编译Hadoop3.2.1源码

最新推荐文章于 2022-04-29 21:16:13 发布

远方的梦想

最新推荐文章于 2022-04-29 21:16:13 发布

阅读量719

点赞数 1

本文链接：https://blog.csdn.net/u011066054/article/details/118049768

版权

java 同时被 3 个专栏收录

4 篇文章 0 订阅

订阅专栏

大数据

1 篇文章 0 订阅

订阅专栏

hadoop

1 篇文章 0 订阅

订阅专栏

1.环境搭建

镜像版本： CentOS-7-x86_64-DVD-1908.iso

SOFTWARE SELECTION：GNOME Desktop（桌面版，比Minimal Install版多部分软件）

Hadoop版本：hadoop-3.2.1-src.tar.gz

下载地址：All previous releases of Hadoop are available from the Apache release archive site.

hadoop源码编译所需的其他软件在源码包的BUILDING.txt文件中有详细介绍，编译过程中发生错误，99%是因为所需软件未安装，根据错误描述安装对应版本软件即可顺利编译。

Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

* Unix System
* JDK 1.8
* Maven 3.3 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* Cyrus SASL devel (if compiling native code)
* One of the compilers that support thread_local storage: GCC 4.8.1 or later, Visual Studio,
  Clang (community version), Clang (version for iOS 9 and later) (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Doxygen ( if compiling libhdfspp and generating the documents )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)

2.软件安装

总结：

①不编译native code和documentation

Create binary distribution without native code and without documentation:

$ mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

则仅需安装JDK 1.8，Maven 3.3 or later，ProtocolBuffer 2.5.0，gcc，gcc-c++，patch即可

②编译native code和documentation

Create binary distribution with native code and with documentation:

$ mvn package -Pdist,native,docs -DskipTests -Dtar

除JDK 1.8，Maven 3.3 or later，ProtocolBuffer 2.5.0，gcc，gcc-c++，patch还需要安装CMake 3.1 or newer，zlib-devel，openssl-devel，cyrus-sasl-devel

1.JDK安装

查看系统自带JDK，自带JDK不满足需要，需卸载自带JDK，自行安装。

[root@bogon ~]# rpm -qa | grep java
tzdata-java-2020a-1.el7.noarch
java-1.8.0-openjdk-headless-1.8.0.262.b10-1.el7.x86_64
java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64
javapackages-tools-3.4.1-11.el7.noarch
java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64
python-javapackages-3.4.1-11.el7.noarch
java-1.7.0-openjdk-headless-1.7.0.261-2.6.22.2.el7_8.x86_64

[root@bogon local]# rpm -e java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64
[root@bogon local]# rpm -e java-1.7.0-openjdk-headless-1.7.0.261-2.6.22.2.el7_8.x86_64
[root@bogon local]# rpm -e --nodeps java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64
[root@bogon local]# rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.262.b10-1.el7.x86_64

#上传并解压
[root@bogon local]# tar -zxvf jdk-8u271-linux-x64.tar.gz

#配置环境变量
[root@bogon local]# vim /etc/profile
#JDK Environment
export JAVA_HOME=/usr/local/jdk1.8.0_271
export PATH=$PATH:$JAVA_HOME/bin

#立即生效
[root@bogon local]# source /etc/profile

#查看JAVA版本
[root@bogon local]# java -version
java version "1.8.0_271"
Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)

2.Maven安装

版本选择：Maven 3.3 or later，下载官网最新版本即可

下载地址：https://maven.apache.org/download.cgi

apache-maven-3.8.1-bin.tar.gz

#上传文件并解压
[root@bogon local]# tar -zxvf apache-maven-3.8.1-bin.tar.gz

#提高依赖包下载速度，修改配置文件，增加阿里镜像或者华为镜像
[root@bogon conf]# vim settings.xml
#<servers>节点增加
    <server>
      <id>huaweicloud</id>
      <username>anonymous</username>
      <password>devcloud</password>
    </server>
#<mirrors>节点增加
    <mirror>
      <id>huaweicloud</id>
      <mirrorOf>*</mirrorOf>
      <name></name>
      <url>https://mirrors.huaweicloud.com/repository/maven</url>
    </mirror>

#配置环境变量
[root@bogon conf]# vim /etc/profile

#Maven Environment
export MAVEN_HOME=/usr/local/apache-maven-3.8.1
export PATH=$PATH:$MAVEN_HOME/bin

#立即生效
[root@bogon conf]# source /etc/profile

#查看Maven版本
[root@bogon conf]# mvn -version
Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
Maven home: /usr/local/apache-maven-3.8.1
Java version: 1.8.0_262, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.el7.x86_64", arch: "amd64", family: "unix"

3.ProtocolBuffer 2.5.0安装

版本选择：ProtocolBuffer 2.5.0

下载地址：https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0

protobuf-2.5.0.tar.gz

ProtocolBuffer安装依赖CMake，gcc，gcc-c++

#上传并解压
[root@bogon local]# tar -zxvf protobuf-2.5.0.tar.gz

#依赖安装
[root@bogon protobuf-2.5.0]# yum install gcc
[root@bogon protobuf-2.5.0]# yum install gcc-c++

#编译安装，如果没有configure则需要运行autogen.sh，autogen.sh可能缺少gtest，需自行下载
[root@bogon protobuf-2.5.0]# ./configure
[root@bogon protobuf-2.5.0]# make
[root@bogon protobuf-2.5.0]# make install

#查看版本
[root@bogon protobuf-2.5.0]# protoc --version
libprotoc 2.5.0

4.CMake(if compiling native code)

版本选择：3.1 or newer

#上传并解压
[root@bogon local]# tar -zxvf cmake-3.20.4.tar.gz

#依赖安装
#openssl-devel安装依赖zlib-devel
[root@bogon cmake-3.20.4]# yum install openssl-devel
[root@bogon cmake-3.20.4]# make
[root@bogon cmake-3.20.4]# make install

#编译安装
[root@bogon cmake-3.20.4]# ./bootstrap
或者
[root@bogon cmake-3.20.4]# ./configure

#查看版本，如果未生效，刷新/etc/profile文件，source /etc/profile
[root@bogon ~]# cmake -version
cmake version 3.20.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).

5.安装Patch

#yum方式安装
[root@bogon hadoop-3.2.1-src]# yum install patch

6.安装Cyrus SASL devel (if compiling native code)

#yum方式安装
[root@bogon hadoop-3.2.1-src]# yum install cyrus-sasl-devel

3.源码编译

仅安装JDK和Maven情况下编译，查看编译错误再依次安装所需软件，详情查看BUILDING.txt文件

#上传文件并解压
[root@bogon local]# tar -zxvf hadoop-3.2.1-src.tar.gz

#编译命令，[]为可选参数

Building distributions:

 * Build distribution with native code    : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar][-Dmaven.javadoc.skip=true]

 Build options:

  * Use -Pnative to compile/bundle native code
  * Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist)
  * Use -Psrc to create a project source TAR.GZ
  * Use -Dtar to create a TAR with the distribution (using -Pdist)
  * Use -Preleasedocs to include the changelog and release docs (requires Internet connectivity)
  * Use -Pyarn-ui to build YARN UI v2. (Requires Internet connectivity)
  * Use -DskipShade to disable client jar shading to speed up build times (in
    development environments only, not to build release artifacts)

#建议跳过测试用例，可快速完成编译
Building distributions:

Create binary distribution without native code and without documentation:

  $ mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Create binary distribution with native code and with documentation:

  $ mvn package -Pdist,native,docs -DskipTests -Dtar

Create source distribution:

  $ mvn package -Psrc -DskipTests

Create source and binary distributions with native code and documentation:

  $ mvn package -Pdist,native,docs,src -DskipTests -Dtar

Create a local staging version of the website (in /tmp/hadoop-site)

  $ mvn clean site -Preleasedocs; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

Note that the site needs to be built in a second pass after other artifacts.

错误1，使用自带JDK1.8

错误信息：[ERROR] Failed to execute goal on project hadoop-annotations: Could not resolve dependencies for project org.apache.hadoop:hadoop-annotations:jar:3.2.1: Could not find artifact jdk.tools:jdk.tools:jar:1.8 at specified path /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-1.el7.x86_64/jre/../lib/tools.jar -> [Help 1]

错误2，缺少ProtocolBuffer 2.5.0

错误信息：[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.2.1:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1]

错误3，缺少patch

错误信息：[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (pre-site) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "patch" (in directory "/usr/local/hadoop-3.2.1-src/hadoop-common-project/hadoop-common"): error=2, No such file or directory
[ERROR] around Ant part ...<exec input="dev-support/jdiff-workaround.patch" dir="/usr/local/hadoop-3.2.1-src/hadoop-common-project/hadoop-common" executable="patch">... @ 4:141 in /usr/local/hadoop-3.2.1-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml

错误4，缺少cmake

错误信息：[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.2.1:cmake-compile (cmake-compile) on project hadoop-common: Error executing CMake: Cannot run program "cmake" (in directory "/usr/local/hadoop-3.2.1-src/hadoop-common-project/hadoop-common/target/native"): error=2, No such file or directory -> [Help 1]

错误5，缺少SASL library (GSASL (gsasl) or Cyrus SASL (libsasl2)

错误信息：[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.2.1:cmake-compile (cmake-compile) on project hadoop-hdfs-native-client: CMake failed with error code 1 -> [Help 1]

以上错误信息不能定位问题，只能发现是cmake发生错误，具体错误信息在上面[WARNING]

[WARNING] CMake Error at main/native/libhdfspp/CMakeLists.txt:135 (message):
[WARNING] Cound not find a SASL library (GSASL (gsasl) or Cyrus SASL (libsasl2).
[WARNING] Install/configure one of them or define NO_SASL=1 in your cmake call
之前以为错误信息在

[WARNING] See also "/usr/local/hadoop-3.2.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/CMakeFiles/CMakeOutput.log".
[WARNING] See also "/usr/local/hadoop-3.2.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/CMakeFiles/CMakeError.log".

日志文件中，然而里面的内容并不能定位问题。在BUILDING.txt中有相关说明Cyrus SASL devel (if compiling native code)