环境:
Mac OSX: 10.9.5
Hadoop: 2.6.0
Java: 1.7.0_71
Mac OSX 下运行Hadoop老出现以下警告, 其实就是缺少本地库
14/12/05 14:40:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
安装 protobuf
首先安装protobuf, 不然编译Hadoop会报错, 从以下地址下载
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
开始编译并安装
./configure --prefix=/apps/dev/protobuf
make -j4
make install
将protobuf放进环境变量
export PATH=/apps/dev/protobuf/bin:$PATH
export DYLD_LIBRARY_PATH=/apps/dev/protobuf/lib
编译Hadoop
首先确定JAVA_HOME在环境变量中
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
必须首先编译Hadoop, 下载Hadoop 源代码
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
tar xf hadoop-2.6.0.tar.gz
cd hadoop-2.6.0-src
/apps/dev/apache-maven-3.2.2/bin/mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
这时候会遇到以下错误
Exception in thread "main" java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/Classes/classes.jar. Expression: file.exists()
解决办法,
sudo mkdir$JAVA_HOME/Classes
sudo ln -sf$JAVA_HOME/lib/tools.jar$JAVA_HOME/Classes/classes.jar
最后经过10分钟, 输出以下信息就是编译完成
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:26 min
[INFO] Finished at: 2014-12-05T19:13:14+08:00
[INFO] Final Memory: 193M/844M
将native 文件夹拷贝到hadoop目录中
rm -rf hadoop-dist/target/hadoop-2.6.0/lib/native
cp -r hadoop-dist/target/hadoop-2.6.0/lib/native /apps/dev/hadoop-2.6.0/lib
修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh
export HADOOP_OPTS="$HADOOP_OPTS-Djava.net.preferIPv4Stack=true -Djava.library.path=/apps/dev/hadoop-2.6.0/native"
此时再运行一次MapReduce Job, 就不会再出现那句警告了。
参考
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html
http://stackoverflow.com/questions/15745010/org-apache-maven-plugin-mojoexecutionexception-protoc-failure