mac os x 编译hadoop

查阅有关官方介绍 http://wiki.apache.org/hadoop/HowToContribute 中有说明：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持 Cygwin  和 Mac OS X 。搜索后发现已经有人给出了Mac OSX 系统下编译生成本地库的patch，下面详细介绍在Mac OSX 平台下编译Hadoop本地库的方法。

[一]、环境说明：

• Hadoop 2.2.0
• Mac OS X 10.9.1

详细的环境依赖（protoc、cmake 等）参见：Hadoop2.2.0源码编译 （http://www.micmiu.com/opensource/hadoop/hadoop-build-source-2-2-0/）中介绍。

[二]、Mac OSX 编译本地库的步骤：

 1、checkout Hadoop 2.2.0的源码

1	$svn co https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0/

2、patch 相关补丁

官方讨论地址：https://issues.apache.org/jira/browse/HADOOP-9648 里面有详细介绍

补丁下载链接：https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch

1	#切换到hadoop 源码的根目录

2	$wget https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch

3	$patch -p1 < HADOOP-9648.v2.patch

ps：如果要回退patch 执行：patch -RE -p1 < HADOOP-9648.v2.patch 即可。

3、编译本地库

在Hadoop源码的根目录下执行编译本地库命令：

1	$ mvn package -Pdist,native -DskipTests -Dtar

编译成功看到如下日志信息：

1	[INFO] ------------------------------------------------------------------------

2	[INFO] Reactor Summary:

3

[INFO]

4	[INFO] Apache Hadoop Main ................................ SUCCESS [1.511s]

5	[INFO] Apache Hadoop Project POM ......................... SUCCESS [0.493s]

6	[INFO] Apache Hadoop Annotations ......................... SUCCESS [0.823s]

7	[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [0.561s]

8	[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.245s]

9	[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [2.465s]

10	[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [0.749s]

11	[INFO] Apache Hadoop Auth ................................ SUCCESS [0.832s]

12	[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.070s]

13	[INFO] Apache Hadoop Common .............................. SUCCESS [1:00.030s]

14	[INFO] Apache Hadoop NFS ................................. SUCCESS [0.285s]

15	[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.049s]

16	[INFO] Apache Hadoop HDFS ................................ SUCCESS [1:13.339s]

17	[INFO] Apache Hadoop HttpFS .............................. SUCCESS [20.259s]

18	[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [0.767s]

19	[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [0.279s]

20	[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.046s]

21	[INFO] hadoop-yarn ....................................... SUCCESS [0.239s]

22	[INFO] hadoop-yarn-api ................................... SUCCESS [7.641s]

23	[INFO] hadoop-yarn-common ................................ SUCCESS [5.479s]

24	[INFO] hadoop-yarn-server ................................ SUCCESS [0.114s]

25	[INFO] hadoop-yarn-server-common ......................... SUCCESS [1.743s]

26	[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [6.381s]

27	[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [0.259s]

28	[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [0.578s]

29	[INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.303s]

30	[INFO] hadoop-yarn-client ................................ SUCCESS [0.233s]

31	[INFO] hadoop-yarn-applications .......................... SUCCESS [0.062s]

32	[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [0.253s]

33	[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.074s]

34	[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [1.504s]

35	[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [0.242s]

36	[INFO] hadoop-yarn-site .................................. SUCCESS [0.172s]

37	[INFO] hadoop-yarn-project ............................... SUCCESS [1.235s]

38	[INFO] hadoop-mapreduce-client-common .................... SUCCESS [3.664s]

39	[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [0.183s]

40	[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [0.495s]

41	[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [1.296s]

42	[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [0.580s]

43	[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [0.213s]

44	[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [0.344s]

45	[INFO] hadoop-mapreduce .................................. SUCCESS [1.303s]

46	[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [0.257s]

47	[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.925s]

48	[INFO] Apache Hadoop Archives ............................ SUCCESS [0.282s]

49	[INFO] Apache Hadoop Rumen ............................... SUCCESS [0.403s]

50	[INFO] Apache Hadoop Gridmix ............................. SUCCESS [0.283s]

51	[INFO] Apache Hadoop Data Join ........................... SUCCESS [0.197s]

52	[INFO] Apache Hadoop Extras .............................. SUCCESS [0.241s]

53	[INFO] Apache Hadoop Pipes ............................... SUCCESS [8.249s]

54	[INFO] Apache Hadoop OpenStack support ................... SUCCESS [0.492s]

55	[INFO] Apache Hadoop Client .............................. SUCCESS [0.373s]

56	[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.133s]

57	[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [0.439s]

58	[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [0.596s]

59	[INFO] Apache Hadoop Tools ............................... SUCCESS [0.044s]

60	[INFO] Apache Hadoop Distribution ........................ SUCCESS [0.194s]

61	[INFO] ------------------------------------------------------------------------

62	[INFO] BUILD SUCCESS

63	[INFO] ------------------------------------------------------------------------

64	[INFO] Total time: 3:44.266s

65	[INFO] Finished at: Fri Jan 17 10:06:17 CST 2014

66	[INFO] Final Memory: 66M/123M

67	[INFO] ------------------------------------------------------------------------

68	micmiu-mbp:trunk micmiu$

编译通过后可在 <HADOOP源码根目录>/hadoop-dist/target/hadoop-2.2.0/lib/ 目录下看到如下内容：

查看源代码

打印 帮助

1	micmiu-mbp:lib micmiu$ tree

2

.

3	|____.DS_Store

4	|____native

5	| |____libhadoop.1.0.0.dylib

6	| |____libhadoop.a

7	| |____libhadoop.dylib

8	| |____libhadooppipes.a

9	| |____libhadooputils.a

10	| |____libhdfs.0.0.0.dylib

11	| |____libhdfs.a

12	| |____libhdfs.dylib

然后把 上面生成的本地库 copy到部署环境相应的位置，再建立软连接即可：

1	$ls -s libhadoop.1.0.0.dylib libhadoop.so

2	$ls -s libhdfs.0.0.0.dylib libhdfs.so

错误处理

运行

clean install  package -Pdist -P-cbuild  -DskipTests  -Dtar 

报各种错误

1、报错[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-7:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]
修改hadoop-common-project/hadoop-common/pom.xml 文件中，env.JAVA_HOME改为java.home

2、报错

/hadoop-2.2.0-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression (‘void’ and ‘int’)
[exec] if(setnetgrent(cgroup) == 1) {
[exec] ~~~~~~~~~~~~~~~~~~~ ^ ~
[exec] 1 error generated.
[exec] make[2]: *** [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c.o] Error 1
[exec] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
[exec] make: *** [all] Error 2

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2 -> [Help 1]

修改mvn3的配置文件：/opt/local/share/java/maven3/settings.xml
在<mirrors>…</mirrors>里添加国内源：

<mirrors>
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexusosc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
</mirrors>

在<profiles>…</profiles>标签中增加以下内容：
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7<k>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>

注意修改jdk version number

将刚才的maven 配置文件拷贝到当前用户的home目录下:
settings.xml   copy 到  your_hadoop_usr_home/.m2/
cp settings.xml ~/.m2

3、报错[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Fatal error compiling: Error while executing the compiler. InvocationTargetException: Java heap space

分配内存不足,参考如下为maven配置JVM参数: export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”

4、报错 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Compilation failure
[ERROR] Failure executing javac, but could not parse the error:

执行maven clean，然后再

export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”
三、最重要的一点，build your code是使用这个command line(Only for Mac OS):
mvn clean install -P-cbuild

编译之前， 你在hadoop-2.2.0-src目录（/Users/JuneMAC/hadoop/release-2.2.0）下执行
mvn clean install –DskipTests

上面的成功后，执行下面这个，生成安装包
mvn clean install  package -Pdist -P-cbuild  -DskipTests  -Dtar

执行完成后，可以在/Users/JuneMAC/hadoop/release-2.2.0/hadoop-dist/target/
下找到
hadoop-2.2.0.tar.gz
将上面这个编译好的源码包解压到：
/Users/JuneMAC/hadoop/
然后进行相关配置
解压之后的源码包和官网下载下来的源码包相对比，没有lib目录
相关解释：
“Here we use the additional options to stop compiling the native code.
this is the key reason why we need use -P-cbuild option”
上面这个是原因，好像不是很重要。实际上如果指定-Pdist,native 生成native lib 不成功，查阅有关官方介绍发现：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持 Cygwin  和 Mac OS X 。