BUILD HADOOP NATIVE LIBRARIES

BUILD HADOOP NATIVE LIBRARIES

[FROM]http://www.drweiwang.com/build-hadoop-native-libraries/

How to build Hadoop Native Libraries for Hadoop 2.2.0

Because the distributed Hadoop 2.2.0 provides a 32bit libhadoop by default, user has to build the native libraries to avoid those warning messages such as, disabled stack guard of libhadoop.so.

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

13/11/01 10:58:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

For the version 2.4.1, the error is like:

14/07/25 13:49:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

 

The official Hadoop websitehttp://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html gives completely unclear instructions on how to build Hadoop native libraries.

So here are what you should do:

You need all the build tools:

$ sudo apt-get install build-essential
$ sudo apt-get install g++ autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev

$ sudo apt-get install maven

Another prerequisite, protoco buffer: protobuf version 2.5, which can be downloaded from https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz. Download it to the /tmp directory; then,

$ tar xzvf protobuf-2.5.0.tar.gz
$ cd protobuf-2.5.0
$ ./configure --prefix=/usr
$ make
$ make check

$ sudo make install

Having all the tools, we can now build Hadoop native libraries. Assuming you have downloaded the Hadoop 2.2.0 source code, do:

$ tar xzvf hadoop-2.2.0-src.tar.gz
$ cd hadoop-2.2.0-src

$ mvn package -Pdist,native -DskipTests -Dtar

Note: there is a missing dependency in the maven project module that results in a build failure at the hadoop-auth stage. Here is the official bug report and fix is

Index: hadoop-common-project/hadoop-auth/pom.xml
===================================================================
--- hadoop-common-project/hadoop-auth/pom.xml (revision 1543124)
+++ hadoop-common-project/hadoop-auth/pom.xml (working copy)
@@ -54,6 +54,11 @@
</dependency>
<dependency>
<groupId>org.mortbay.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
<scope>test</scope>
</dependency>


Maven will do all the heavy work for you, and you should get this after build is completed

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------
[INFO] Total time: 15:39.705s
[INFO] Finished at: Fri Nov 01 14:36:17 CST 2013
[INFO] Final Memory: 135M/422M
The built native libraries should be at

hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/lib