hadoop编译java,Hadoop源码编译(2.4.1)

背景

Hadoop 2.4.1预编译版本自带的libhadoop.so是在32位机器上编译上,导致在64位OS上运行时，总出现如下告警：

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

查看libhadoop.so的位数，显示是32位的：

file libhadoop.so.1.0.0

libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped

为彻底解决此问题，自行编译Hadoop 2.4.1源码。

编译过程

安装相关软件

yum install -y cmake autoconf automake libtool gcc zlib1g-dev pkg-config libssl-dev openssl gcc g++ make maven zlib zlib1g-dev libcurl4-o

安装protobuf-2.5.0

1

2

3

4

5

6wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz

tar -zxvf protobuf-2.5.0.tar.gz

cdprotobuf-2.5.0

./configure

make

make install

编译Hadoop源码

1

2

3

4

5wget https://archive.apache.org/dist/hadoop/core/hadoop-2.4.1/hadoop-2.4.1-src.tar.gz

tar -zxvf hadoop-2.4.1-src.tar.gz

cdhadoop-2.4.1-src

exportPlatform=x64

mvn package -Pdist,native -DskipTests -Dtar

查看编译后的本地库，已经是64位：

1

2

3cdhadoop-dist/target/hadoop-2.4.1/lib/native

file libhadoop.so.1.0.0

libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

拷贝到Hadoop环境中

1

2

3

4

5cdhadoop-dist/target/hadoop-2.4.1/lib/native

cp * /opt/hadoop/lib/native/

scp * data01:/opt/hadoop/lib/native/

scp * data02:/opt/hadoop/lib/native/

scp * data03:/opt/hadoop/lib/native/

修改环境变量

在etc/hadoop/hadoop-env.sh中增加：

1

2exportLD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/

exportJAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/opt/hadoop/lib/native/

参考文档