CentOS编译Hadoop3.2.1源码

CentOS 8

Hadoop3.2.1

 

BUILDING.txt:

Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

* Unix System
* JDK 1.8
* Maven 3.3 or later
* ProtocolBuffer 2.5.0
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* Cyrus SASL devel (if compiling native code)
* One of the compilers that support thread_local storage: GCC 4.8.1 or later, Visual Studio,
  Clang (community version), Clang (version for iOS 9 and later) (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Doxygen ( if compiling libhdfspp and generating the documents )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)

 

jdk8、maven、cmake、protobuf的安装看这里  ←跟着这个做就好了，我下面写的就简单点

1 .jdk1.8

2. maven3.6.3   我安装在/opt/maven， 配置了环境变量（/etc/profile），添加阿里云镜像源(/opt/maven/conf/settings.xml)

3. protobuf2.5.0 （必须是2.5.0，不能是别的版本）

   yum配置阿里云镜像 https://developer.aliyun.com/mirror/centos?spm=a2c6h.13651102.0.0.3e221b11LaWY2P

   yum安装gcc，make啥的

   然后protobuf解压到/opt/protobuf，添加环境变量

4. cmake3.1   去官网下，yum装好像最高版本没有3.1

    https://www.freesion.com/article/5623174550/

    运行./configure后会提示找不到open ssl，  然而openssl已经有了，但/usr/include里面却没有openssl文件夹，所以
 

   yum install openssl-devel

5. 还有一些是上面教程没有提到要装，但是building.txt里有的，一并装上

yum install autoconf automake libtool cyrus-sasl-devel ncurses-devel

autoconf automake libtool 看到有人说装这几个才不会报错，所以一起装上了

zlib-devel已经有了，不知道是centOS8自带的还是什么时候装上了- - 没有的话也是一样，yum install zlib-devel

 

编译：

下载hadoop3-2.1源码（hadoop3-2-1.tar.gz），解压，开始编译

tar -zxvf hadoop-3.2.1.tar.gz   
cd hadoop-3.2.1
mvn clean package -Pdist,native -DskipTests -Dtar -e

 

报错

1.java.lang.OutOfMemoryError: Java heap space

解决办法：命令行输入

export HADOOP_CLIENT_OPTS="-Xmx2048m" 

再重新编译

 

2.Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)

安装doxygen  具体见 → https://www.doxygen.nl/download.html  （注意build的话，需要flex等工具，要先安装）

 

编译完成后，会出现hadoop-dist目录，hadoop-dist/target里面的hadoop-3.2.1.tar.gz就可以拿去部署了。或者用hadoop-dist/share里面的jar包替换原来集群的share目录

 

参考：

https://johngene.cn/archives/469

https://blog.csdn.net/Amber_amber/article/details/41041787

https://blog.csdn.net/forevernull/article/details/28656777