Apache官方网站提供了32bit的Hadoop2.2.0可执行文件,对于时下主流的64bit操作系统来说无疑是一个小麻烦。由于本人是新手,编译过程种遇到不少问题,折腾两天终于编译成功,在此记录下编译过程(大家有兴趣的话可以尝试编译,也可以选择直接下载已编译好的64bit版本,提供几个网址)。
http://pan.baidu.com/s/1hqomHik
http://pan.baidu.com/s/1kTjuKflhttp://pan.baidu.com/share/link?shareid=956286972&uk=2032187724 密码:n0kf另外提供几个关于hadoop-2.2.0 64位编译介绍的链接
http://www.chepoo.com/hadoop2-2-0-64-the-digital-source-compile-notes.htmlhttp://blog.csdn.net/jiutianhe/article/details/17691629http://ju.outofmemory.cn/entry/64668http://wenku.baidu.com/link?url=6ohT0scYmX8g72jX7xr8q6Kg_IwmgGAp7jgDH_GQ7A_p1w2iA8TH7xSvvFRDKgyWee37svAfYuTKBm1L2nt4HtMDdRTqU26IDjzTW8eDzFu
编译环境配置
首先是一些库文件,编译工具的安装1.jdk1.7.0
http://www.oracle.com/technetwork/java/javase/downloads/index.html2.maven 不要使用最新的maven3.1.1版本,与hadoop2.2.0存在兼容问题,使用maven3.0.0
http://maven.apache.org/download.cgi3.ant-1.9.2
http://ant.apache.org/bindownload.cgi4.findbugs 2.0.2
http://sourceforge.jp/projects/sfnet_findbugs/releases5.protobuf 2.5.0
https://code.google.com/p/protobuf/downloads/list以上文件除mvn和protoc外,其他文件直接解压,然后设置环境变量即可使用。
mvn安装,由于maven的国外服务器可能连接不上,而且就算连接上,其下载速度也足够令人窒息,所以此处首先需要配置国内镜像。在maven目录下,conf/settings.xml,在<mirrors></mirros>里添加,原本的不要动
<mirror> <id>nexus-osc</id> <mirrorOf>*</mirrorOf> <name>Nexusosc</name> <url>http://maven.oschina.net/content/groups/public/</url> </mirror>
在<profiles></profiles>内新添加,原本的不要动
<profile> <id>jdk-1.7</id> <activation> <jdk>1.4</jdk> </activation> <repositories> <repository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> <pluginRepositories> <pluginRepository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </pluginRepository> </pluginRepositories> </profile>
protoc2.5.0需要经过编译才能安装,进入protobuf 解压包,
./cinfig make make install
以root用户设置环境变量 (根据自己的安装目录进行设置,这里除jdk外,都安装在 /usr/local 目录下)
vi /etc/profile export JAVA_HOME=/usr/java/jdk1.7.0 export PATH=$JAVA_HOME:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export MAVEN_HOME=/usr/local/apache-maven-3.0.5/bin export PATH=$MAVEN_HOME:$PATH export ANT_HOME=/usr/local/ant-1.9.2/ export PATH=$ANT_HOME/bin:$PATH export FINDBUGS_HOME=/usr/local/findbugs-2.0.2/bin export PATH=$FINDBUGS_HOME:$PATH export PROTOC_HOME=/usr/local/protobuf-2.5.0 export PATH=$PROTOC_HOME/bin:$PATH export LD_LIBRARY_PATH=LD_LIBRARY_PATH:/usr/local/lib
wq保存退出,运行下列命令是更改生效
source /etc/profile
验证是否安装成功
mvn -version ant -version java -version protoc --version(注意 是-- 而非- ) findbugs -version
若安装未成功,适当调整环境变量设置,应该没问题。
直接命令行安装其他辅助编译文件
yum -y install svn ncurses-devel gcc* yum -y install lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel
至此,编译环境基本设置成功,
hadoop2.2.0源文件下载,下载hadoop-2.2.0-src.tar.gz 19M
ftp://ftp.osuosl.org/pub/apache/hadoop/common/hadoop-2.2.0/
最新的Hadoop 2.2.0 的Source Code 压缩包解压出来的code有个bug 需要patch后才能编译,
https://issues.apache.org/jira/browse/HADOOP-10110
下载后,进入hadoop 解压包执行下列命令
patch -p0 < HADOOP-10110
会出现执行成功的提示。
现在可以开始编译hadoop-2.2.0.src.tar.gz 了,进入hadoop文件夹,执行下列命令
mvn package -DskipTests -Pdist,native -Dtar
时长在一个小时左右,如果执行时间太长的话,一是网速实在不行,二是maven 没有设置国内镜像,三是要出现error。出现error也不用担心,仔细检查提示,容易出现问题有,maven没有设置国内镜像;hadoop没有设置patch;多次运行出错之后仍然继续执行编译命令,此时会出现jdk运行出错的提示,RuntimeException提示,所以重新编译前都应该先执行命令,删除原先编译过程中产生的冗余文件,最好重新解压编译。
mvn clean
这里有几个常见问题帖子,大家可以参考下
http://blog.csdn.net/amaowolf/article/details/8125351
http://blog.sina.com.cn/s/blog_6baac06c0101gvxr.html
http://www.superwu.cn/2013/12/26/913/
编译成功后会出现如下画面
- [INFO] ------------------------------------------------------------------------
- [INFO] Reactor Summary:
- [INFO]
- [INFO] Apache Hadoop Main ................................ SUCCESS [3.709s]
- [INFO] Apache Hadoop Project POM ......................... SUCCESS [2.229s]
- [INFO] Apache Hadoop Annotations ......................... SUCCESS [5.270s]
- [INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.388s]
- [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [3.485s]
- [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [8.655s]
- [INFO] Apache Hadoop Auth ................................ SUCCESS [7.782s]
- [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [5.731s]
- [INFO] Apache Hadoop Common .............................. SUCCESS [1:52.476s]
- [INFO] Apache Hadoop NFS ................................. SUCCESS [9.935s]
- [INFO] Apache Hadoop Common Project ...................... SUCCESS [0.110s]
- [INFO] Apache Hadoop HDFS ................................ SUCCESS [1:58.347s]
- [INFO] Apache Hadoop HttpFS .............................. SUCCESS [26.915s]
- [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [17.002s]
- [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [5.292s]
- [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.073s]
- [INFO] hadoop-yarn ....................................... SUCCESS [0.335s]
- [INFO] hadoop-yarn-api ................................... SUCCESS [54.478s]
- [INFO] hadoop-yarn-common ................................ SUCCESS [39.215s]
- [INFO] hadoop-yarn-server ................................ SUCCESS [0.241s]
- [INFO] hadoop-yarn-server-common ......................... SUCCESS [15.601s]
- [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [21.566s]
- [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [4.754s]
- [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [20.625s]
- [INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.755s]
- [INFO] hadoop-yarn-client ................................ SUCCESS [6.748s]
- [INFO] hadoop-yarn-applications .......................... SUCCESS [0.155s]
- [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [4.661s]
- [INFO] hadoop-mapreduce-client ........................... SUCCESS [0.160s]
- [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [36.090s]
- [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [2.753s]
- [INFO] hadoop-yarn-site .................................. SUCCESS [0.151s]
- [INFO] hadoop-yarn-project ............................... SUCCESS [4.771s]
- [INFO] hadoop-mapreduce-client-common .................... SUCCESS [24.870s]
- [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [3.812s]
- [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [15.759s]
- [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [6.831s]
- [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [8.126s]
- [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [2.320s]
- [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [9.596s]
- [INFO] hadoop-mapreduce .................................. SUCCESS [3.905s]
- [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [7.118s]
- [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [11.651s]
- [INFO] Apache Hadoop Archives ............................ SUCCESS [2.671s]
- [INFO] Apache Hadoop Rumen ............................... SUCCESS [10.038s]
- [INFO] Apache Hadoop Gridmix ............................. SUCCESS [6.062s]
- [INFO] Apache Hadoop Data Join ........................... SUCCESS [4.104s]
- [INFO] Apache Hadoop Extras .............................. SUCCESS [4.210s]
- [INFO] Apache Hadoop Pipes ............................... SUCCESS [9.419s]
- [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [2.306s]
- [INFO] Apache Hadoop Tools ............................... SUCCESS [0.037s]
- [INFO] Apache Hadoop Distribution ........................ SUCCESS [21.579s]
- [INFO] Apache Hadoop Client .............................. SUCCESS [7.299s]
- [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [7.347s]
- [INFO] ------------------------------------------------------------------------
- [INFO] BUILD SUCCESS
- [INFO] ------------------------------------------------------------------------
- [INFO] Total time: 11:53.144s
- [INFO] Finished at: Fri Nov 22 16:58:32 CST 2013
- [INFO] Final Memory: 70M/239M
- [INFO] ------------------------------------------------------------------------
编译后的路径在:hadoop-2.2.0-src/hadoop-dist/target/
- [root@localhost bin]# ./hadoop version
- Hadoop 2.2.0
- Subversion Unknown -r Unknown
- Compiled by root on 2013-11-22T08:47Z
- Compiled with protoc 2.5.0
- From source with checksum 79e53ce7994d1628b240f09af91e1af4
- This command was run using /data/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar
可以看出hadoop的版本
- [root@localhost hadoop-2.2.0]# file lib//native/*
- lib//native/libhadoop.a: current ar archive
- lib//native/libhadooppipes.a: current ar archive
- lib//native/libhadoop.so: symbolic link to `libhadoop.so.1.0.0'
- lib//native/libhadoop.so.1.0.0: <span style="color:#ff0000;">ELF 64-bit LSB shared object, x86-64, version 1</span> (SYSV), dynamically linked, not stripped
- lib//native/libhadooputils.a: current ar archive
- lib//native/libhdfs.a: current ar archive
- lib//native/libhdfs.so: symbolic link to `libhdfs.so.0.0.0'
- lib//native/libhdfs.so.0.0.0: <span style="color:#ff0000;">ELF 64-bit LSB shared object, x86-64, version 1</span> (SYSV), dynamically linked, not stripped
注意红色字体部分,如果下载官网的编译好的包,这里显示的是32-bit。
只为共同学习,欢迎大家相互讨论!