Mac OSX 下 Hadoop 使用本地库提高效率

最新推荐文章于 2023-07-17 17:44:17 发布

Hedgelion

最新推荐文章于 2023-07-17 17:44:17 发布

阅读量614

点赞数

分类专栏： hadoop 文章标签： hadoop

hadoop 专栏收录该内容

0 篇文章 0 订阅

订阅专栏

环境：

Mac OSX: 10.9.5
Hadoop: 2.6.0
Java: 1.7.0_71

Mac OSX 下运行Hadoop老出现以下警告, 其实就是缺少本地库

14/12/05 14:40:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

安装 protobuf

首先安装protobuf, 不然编译Hadoop会报错, 从以下地址下载

https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

开始编译并安装

./configure --prefix=/apps/dev/protobuf
make -j4
make install

将protobuf放进环境变量

export PATH=/apps/dev/protobuf/bin:$PATH
export DYLD_LIBRARY_PATH=/apps/dev/protobuf/lib

编译Hadoop

首先确定JAVA_HOME在环境变量中

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home

必须首先编译Hadoop, 下载Hadoop 源代码

http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

tar xf hadoop-2.6.0.tar.gz
cd hadoop-2.6.0-src
/apps/dev/apache-maven-3.2.2/bin/mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true

这时候会遇到以下错误

Exception in thread "main" java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()

解决办法,

sudo mkdir $JAVA_HOME/Classes
sudo ln -sf $JAVA_HOME/lib/tools.jar $JAVA_HOME/Classes/classes.jar

最后经过10分钟, 输出以下信息就是编译完成

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:26 min
[INFO] Finished at: 2014-12-05T19:13:14+08:00
[INFO] Final Memory: 193M/844M

将native 文件夹拷贝到hadoop目录中

rm -rf hadoop-dist/target/hadoop-2.6.0/lib/native
cp -r hadoop-dist/target/hadoop-2.6.0/lib/native /apps/dev/hadoop-2.6.0/lib

修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.library.path=/apps/dev/hadoop-2.6.0/native"

此时再运行一次MapReduce Job, 就不会再出现那句警告了。

参考

http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html
http://stackoverflow.com/questions/15745010/org-apache-maven-plugin-mojoexecutionexception-protoc-failure