Hadoop3 - 源码编译

环境

OS: Centos-Stream 8

JDK: openjdk1.8

下载源码

https://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4-src.tar.gz

安装依赖

编译依赖

maven 3.9.0

操作系统依赖

yum install bzip2 clang cmake gcc gcc-c++ python39 fuse3 flex bison snappy automake cyrus-sasl cyrus-sasl-devel openssl openssl-devel git libtirpc libtirpc-devel

其他依赖

1. 安装protobuf2.5

wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
# 解压缩
tar -zxvf protobuf-2.5.0.tar.gz 
cd protobuf-2.5.0/
./configure --prefix=/opt/app/protobuf
make
make install
# /etc/profile.d/proto.sh 
export PROTOBUF_HOME=/opt/app/protobuf
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${PROTOBUF_HOME}/lib
export LIBRARY_PATH=$LIBRARY_PATH:${PROTOBUF_HOME}/lib
export PATH=$PATH:${PROTOBUF_HOME}/bin

2. doxygen 依赖

wget https://github.com/doxygen/doxygen/archive/refs/tags/Release_1_9_6.tar.gz

 mkdir build
 cd build
cmake -G "Unix Makefiles" ..
make
make install

代码调整

修改hadoop-common-project/hadoop-common/pom.xml文件增加依赖

    <dependency>
      <groupId>javax.ws.rs</groupId>
      <artifactId>jsr311-api</artifactId>
    </dependency>

编译

编译代码： mvn clean package -DskipTests -Pdist,native -Dtar