os:centos 6.7 x64
cdh5.11.1
要解决的问题:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解决的必要性
hadoop的cache和短路读(Short-Circuit Local Reads)都需要native library的支持
解决步骤
编译方法是
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/NativeLibraries.html
============
编译过程中出现
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.3.0:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1]
这时需要在系统中安装protobuf
从https://github.com/google/protobuf/archive/v3.2.0.tar.gz下载3.2.0源码3.3.0编译有问题,
1、 下载工具:
$ sudo yum install autoconf automake libtool curl make g++ unzip
2、 首先生成configure script
$ ./autogen.sh
3、 编译和安装
$ ./configure
出错:configure: error: C++ preprocessor "/lib/cpp" fails sanity check
解决:这是缺少相应的库导致的,安装相应的库也可。
# yum install glibc-headers
# yum install gcc-c++
再执行$ ./configure 就OK了。
4、 继续编译
$ make
出错:
./google/protobuf/metadata.h:175: 错误:类‘google::protobuf::internal::InternalMetadataWithArena’ 没有名为‘InternalMetadataWithArenaBase’ 的字段
./google/protobuf/metadata.h:204: 错误:类‘google::protobuf::internal::InternalMetadataWithArenaLite’ 没有名为‘InternalMetadataWithArenaBase’ 的字段
解决:把src/google/protobuf/metadata.h:175和204:的代码改一下(参考https://github.com/google/protobuf/pull/2599/commits/141a1dac6ca572056c6a8b989e41f6ee213f8445#diff-4842737ed3a65a110689915a0540b272L175)
再$ make,OK
5、 继续make check,时间确实好长
$ make check
$ sudo make install
$ sudo ldconfig # refresh shared library cache.
即可.
6、 测试一下
$ protoc –version
返回libprotoc 3.2.0
安装OK
好不容易安装了3.2.0,再便宜hadoop-common,居然居然
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.6.0-cdh5.11.1:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 3.2.0', expected version is '2.5.0'
我XXXXXXXXXX啊!!!!!!!!!!!!!
$ make uninstall #卸载了protoc
下载了protobuf 2.5.0,编译,提示Fetching gtest-1.5.0 from the web...
curl: (7) Failed to connect to 2404:6800:4008:c04::52: 网络不可达
$ git clone https://github.com/google/googletest.git下载gtest1.5
$ git tag
$ git checkout release-1.5.0
$ export GTEST_DIR=/data/sftp/mysftp/upload/gtest1.5/googletest
$ g++ -I${GTEST_DIR}/include -I${GTEST_DIR} -c ${GTEST_DIR}/src/gtest-all.cc
$ ar -rv libgtest.a gtest-all.o
$ cp *.a /usr/local/lib
这样gtest装上了,但是protobuf的./autogen.sh还是报错:
下载了protobuf 2.5.0,编译,提示Fetching gtest-1.5.0 from the web...
curl: (7) Failed to connect to 2404:6800:4008:c04::52: 网络不可达
干脆从http://download.csdn.net/download/shingo0109/8324655下了一个gtest-1.5.0.tar.gz,然后如下操作:
$ tar -vxzf gtest-1.5.0.tar.gz
$ mv gtest-1.5.0 protobuf-2.5.0/gtest
再执行./autogen.sh,顺利通过。
l 再继续执行:
$ ./configure
$ make
$ make check #出了一堆ERROR,先不管他
$ sudo make install
$ sudo ldconfig # refresh shared library cache.
$ protoc --version
libprotoc 2.5.0
l 下面编译hadoop native lib
$ cd /usr/hadoop/hadoop-2.6.0-cdh5.11.1/src/hadoop-common-project/
$ mvn package -DskipTests -Pdist,native,docs –Dtar
l 又出错:
[WARNING] [svn, info] failed: java.io.IOException: Cannot run program "svn": error=2, 没有那个文件或目录
[WARNING] [git, branch] failed with error code 128
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec dir="/usr/hadoop/hadoop-2.6.0-cdh5.11.1/src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failοnerrοr="true">... @ 4:142 in /usr/hadoop/hadoop-2.6.0-cdh5.11.1/src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
可能少了几个包,安装一下:
$ yum –y install zlib-devel
$ yum -y install openssl-devel
还少svn,继续:
$ rpm -qa subversion #看一下有没有安装svn
$ yum list | grep subversion
$ yum -y install subversion-devel.x86_64
$ wget http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.9-bin.tar.gz
$ tar -zxvf apache-ant-1.9.9-bin.tar.gz
$ mv apache-ant-1.9.9 /usr/local/ant
$ vim /etc/profile
export ANT_HOME=/usr/local/ant
export PATH=$PATH:$ANT_HOME/bin
$ ant # 测试一下ok
l 继续编译,有错误就重新来一遍
$ cd /usr/hadoop/hadoop-2.6.0-cdh5.11.1/src/hadoop-common-project/
$ mvn package -DskipTests -Pdist,native,docs –Dtar
SUCCESS!!!
编译好的文件在/usr/hadoop/hadoop-2.6.0-cdh5.11.1/src/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib
把他们拷贝到native目录下:
$ cp * /usr/hadoop/hadoop-2.6.0-cdh5.11.1/lib/native/
测试一下hadoop:
$ Hadoop fs –ls
不报没有本地lib的错了!