ARM下编译HDP 3.1.5-Hadoop 3.1.1

1.环境信息

项目	说明
服务器	FT-2000+服务器
CPU	Phytium,FT2000PLUS
操作系统	Kylin 4.0.2
内核	4.4.131-20190726.kylin.server-generic

 

 

 

 

 

 

2.配置编译环境

2.1安装基本库

apt-get install wget patch automake libtool make cmake git gcc openssl libssl-dev zlib1g zlib1g.dev libc6-dev g++

2.2安装OpenJDK

apt-get install -y openjdk-8-jdk

echo "

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64

export PATH=$PATH:$JAVA_HOME/bin

" >> /etc/profile

source /etc/profile

查看版本信息：java -version

附手动安装下载地址：https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u222-b10/OpenJDK8U-jdk_aarch64_linux_hotspot_8u222b10.tar.gz

2.3安装Maven

下载：https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz

unzip apache-maven-3.5.4.zip -d /data/module

echo "

export MAVEN_HOME=/data/module/apache-maven-3.5.4

export PATH=$PATH:$MAVEN_HOME/bin

" >> /etc/profile

source /etc/profile

查看版本信息：mvn -v

添加华为源和hortonworks源

vim settings.xml

<mirror>
  <id>jetty</id>
  <mirrorOf>jetty Repository</mirrorOf>
  <name>howntework jetty Repository</name>
  <url>http://repo.hortonworks.com/content/groups/public/</url>
</mirror>

<mirror>
  <id>huaweimaven</id>
  <name>huawei maven</name>
  <url>https://mirrors.huaweicloud.com/repository/maven/</url>
  <mirrorOf>central</mirrorOf>
</mirror>

2.4安装cmake

Hadoop编译要求3.12及以上版本，建议用3.12.4。

下载：https://cmake.org/files/v3.12/cmake-3.12.4.tar.gz

tar -zxf cmake-3.12.4.tar.gz

cd cmake-3.12.4

./bootstrap

make -j8

make install

查看版本信息：cmake --version

2.5安装protoc

建议用2.5.0版本。

下载：https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz

//安装依赖库

sudo apt-get install patch automake libtool

tar -zxf protobuf-2.5.0.tar.gz

patch补丁下载路径：

https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/bigdata/Patch/protoc.zip

//上传protoc.patch到服务器，打补丁，其中protoc.patch的路径视实际情况而定

cp protoc.patch ./src/google/protobuf/stubs/

cd ./src/google/protobuf/stubs/

patch -p1 < protoc.patch

//编译并安装到系统默认目录

./autogen.sh && ./configure CFLAGS='-fsigned-char' && make -j8 && make install

3.编译Hadoop

//从Github上下载HDP3.1 Hadoop源码并解压

wget https://github.com/hortonworks/hadoop-release/archive/HDP-3.1.5.0-152-tag.tar.gz

tar -zxvf hadoop-release-HDP-3.1.5.0-152-tag.tar.gz

//进入hadoop源码解压目录，修改hadoop-project/pom.xml，在文件第1976行开始，修改和添加maven仓库源

cd hadoop-release-HDP-3.1.5.0-152-tag

vim hadoop-project/pom.xml

<repositories>

<repository>

<id>dynamodb-local-oregon</id>

<name>DynamoDB Local Release Repository</name>

<!--<url>https://s3-us-west-2.amazonaws.com/dynamodb-local/release</url>-->

<url>https://repo.hortonworks.com/content/repositories/releases</url>

</repository>

<repository>

<id>kunpengmaven</id>

<name>kunpeng maven</name>

<url>https://mirrors.huaweicloud.com/kunpeng/maven</url>

</repository>

<repository>

<id>amazonmaven</id>

<name>amazonmaven</name>

<url>https://repository.mulesoft.org/nexus/content/repositories/public</url>

</repository>

</repositories>

//修改hadoop-hdfs-project/hadoop-hdfs/pom.xml，在文件底部，添加maven仓库源

vim hadoop-hdfs-project/hadoop-hdfs/pom.xml

<pluginRepositories>

<pluginRepository>

<id>hortonworkmaven</id>

<name>hortonwork maven</name>

<url>https://repo.hortonworks.com/content/repositories/releases</url>

</pluginRepository>

</pluginRepositories>

//修改hadoop-common-project/hadoop-common/pom.xml，将两处${env.JAVA_HOME}修改为实际JDK安装路径

vim hadoop-common-project/hadoop-common/pom.xml

<!--<javahPath>${env.JAVA_HOME}/bin/javah</javahPath>-->

<javahPath>/usr/lib/jvm/java-8-openjdk-arm64/bin/javah</javahPath>

//安装snappy，支持snappy压缩

tar xf snappy-1.1.3.tar.gz

./configure && make && make install

默认安装在/usr/local/lib,在其下会有snappy相关.so结尾的库文件

//执行编译，其中-Dsnappy.lib参数值设为libsnappy.so所在的目录

cd /data/software/hadoop-release-HDP-3.1.5.0-152-tag

export MAVEN_OPTS="-Xms256m -Xmx512m"

mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy -Dmaven.javadoc.skip=true

编译完成，在hadoop-dist/target目录下生成tar.gz包

4.故障排除

在编译hadoop-common过程中报错：

JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND

Failed to find a viable JVM installation under JAVA_HOME

cmake-compile (cmake-compile) on project hadoop-common:CMake failed with error code.

解决：

网上找了很久，有的说升级cmake版本，有的说JDK配置有问题，尝试了很多办法，都没有解决。

如下两个参数对应的文件应该是这样：

 JAVA_AWT_LIBRARY=/usr/lib/jvm/java-8-openjdk-arm64/jre/lib/aarch64/libawt.so

JAVA_JVM_LIBRARY=/usr/lib/jvm/java-8-openjdk-arm64/jre/lib/aarch64/server/libjvm.so

最后怀疑是内核版本问题，在HadoopJNI.cmake下添加如下两行，居然编译成功了。这里是个比较坑的地方，之前编译HDP3.1.0没有出现这样的问题。

    elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64")
        set(_java_libarch "aarch64")