CentOS7.7编译hadoop3.1.3

1、准备工作

1.JDK下载  最低版本在1.8
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

2.hadoop3.1.3源码
https://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/

3.maven
http://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/

4.protobuf （序列化框架）
https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0

5.cmak(构建项目工具)
https://cmake.org/files/v3.17/

6.ant （java生成器)
https://downloads.apache.org/ant/binaries/

7.findbug(代码检查工具)
https://sourceforge.net/projects/findbugs/

8.isa-l 加速库
https://github.com/intel/isa-l

测试网络是否连通
ping www.baidu.com

2、工具包安装

2.1、创建路径

mkdir -p /ayuan/software/haddop_source

mkdir -p /ayuan/service/hadoop_source

上传软件包

解压软件包到 /ayuan/service/hadoop_source

1.解压JDK

tar -zxvf jdk-8u241-linux-x64.tar.gz -C /ayuan/service/hadoop_source/

2.解压hadoop源码

 tar -zxvf hadoop-3.1.3-src.tar.gz -C /ayuan/service/hadoop_source/

3.解压maven

tar -zxvf apache-maven-3.6.3-bin.tar.gz -C /ayuan/service/hadoop_source/      

4.解压protobuf

tar -zxvf protobuf-2.5.0.tar.gz -C /ayuan/service/hadoop_source/    

5.解压cmake

tar -zxvf cmake-3.17.0.tar.gz -C /ayuan/service/hadoop_source/    

注意：

cd  /ayuan/service/hadoop_source/hadoop-3.1.3-src/hadoop-hdfs-project/hadoop-hdfs/src/main/resources

更改hdfs副本的数量为3   查询位置为 dfs.replication

2.2、安装JDK

1.解压JDK  前面已经解压过了
tar -zxvf jdk-8u241-linux-x64.tar.gz -C /ayuan/service/hadoop_source/

2.配置环境变量
vim /etc/profile

3.刷新环境变量并查看jdk是否装好
source /etc/profile
java -version
java
javac

2.3、安装maven

1.配置maven环境变量
vim /etc/profile
#MAVEN_HOME
MAVEN_HOME=/ayuan/service/hadoop_source/apache-maven-3.6.3
PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin

source /etc/profile

2.修改maven的镜像

vim conf/settings.xml

    <mirror>
      <id>nexus-aliyun</id>
      <mirrorOf>central</mirrorOf>
      <name>Nexus aliyun</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public</url>
    </mirror>
 
 3.验证maven是否成功
 mvn -version

2.4、安装依赖

1.安装gcc make 
yum install -y gcc* make autoconf automake libtool yasm git 

2.安装压缩工具
yum -y install snappy*  bzip2* lzo* zlib*  lz4* gzip*

3.安装一些基本工具
yum -y install openssl* svn ncurses* autoconf automake libtool

4.安装扩展源，才可安装zstd
yum -y install epel-release

5.安装zstd
yum -y install *zstd*

2.5、安装cmake

1.执行安装：
cd /ayuan/service/hadoop_source/cmake-3.17.0
./bootstrap
make && make install 
2.验证是否安装成功
cmake -version

2.6、安装protobuf

1.指定安装：
cd /ayuan/service/hadoop_source/protobuf-2.5.0

./configure --prefix=/opt/module/hadoop_source/protobuf-2.5.0
make && make install

2.配置环境变量
#PROTOC_HOME
PROTOC_HOME=/ayuan/service/hadoop_source/protobuf-2.5.0
PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin:$PROTOC_HOME/bin

3.验证
source /etc/profile
protoc --version

2.7 isa-l加速器安装

yum install -y yasm
yum install -y nasm

git clone https://github.com/01org/isa-l/

./autogen.sh
/configure --prefix=/ayuan/service/hadoop_source/isa-l --libdir=/ayuan/service/hadoop_source/isa-l/lib64
./configure --prefix=/usr --libdir=/usr/lib64

make && make install 

2.8、安装ant（可不装）

1.配置环境变量
#ant
export ANT_HOME=/ayuan/service/hadoop_source/apache-ant-1.10.12
export PATH=$ANT_HOME/bin:$PATH

source /etc/profile
ant -version

2.9、安装findbug(可不装)

1.配置环境变量
#findbug
export FINDBUGS_HOME=/ayuan/service/hadoop_source/findbugs-3.0.1
export PATH=$FINDBUGS_HOME/bin:$PATH

source /etc/profile
ant -version

3、源码编译

cd /ayuan/service/hadoop_source/hadoop-3.1.3-src

mvn clean package -DskipTests -Pdist,native -Dtar

大概需要2-3个小时编译，主要看网速

build sucess  即代表成功

编译成功的包在
/ayuan/service/hadoop_source/hadoop-3.1.3-src/hadoop-dist/target