由于现在hadoop2.0还处于beta版本,在apache官方网站上发布的beta版本中只有编译好的32bit可用,如果你直接下载安装在64bit的linux系统的机器上,运行会报一个INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable的错误,但在实际测试中是可以正常安装并可以运行自带的wordcont例子,所以这个错误不会导致hadoop2.1.0的安装和运行失败,此错误引起原因是由于鉴于性能问题以及某些Java类库的缺失,对于某些组件,Hadoop提供了自己的本地实现。 这些组件保存在Hadoop的一个独立的动态链接的库里。这个库在*nix平台上叫libhadoop.so,此文件在发行的hadoop版本lib/native目录下。详细请看下面的这个官方链接hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-common/NativeLibraries.html。当然这里也可以在core-site.xml里配置不使用本地库。
一、准备环境(先列出这些所需的软件环境,如果不安装除hadoop2.1.0源码中BUILDING.txt要求的软件环境则编译的时候会报错,后面会加以说明):
1.CentOS-6.4-x86_64-bin-DVD1(1).iso(不一定使用centos系统,主流的unix like即可,但后面软件环境可能有出入)
下载地址:isoredirect.centos.org/centos/6/isos/x86_64/
2.hadoop-2.1.0-beta-src.tar.gz
下载地址:apache.dataguru.cn/hadoop/common/hadoop-2.1.0-beta/
下载地址:svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta/(svn工具checkout链接)
上面两个链接都可以获取hadoop2.1.0-beta源码
3.jdk-7u40-linux-x64.rpm(hadoop2.1.0源码中BUILDING.txt中要求使用1.6 or newer)
下载地址:www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
4.apache-maven-3.0.5-bin.tar.gz(hadoop2.1.0源码中BUILDING.txt中要求使用3.0,从hadoop2.0版本以后使用maven编译,之前用Ant)
下载地址:maven.apache.org/download.cgi(目前最新版本3.1,hadoop2.1.0源码中BUILDING.txt中要求使用3.0)
5.apache-ant-1.9.2-bin.zip(下载二进制版本的,这个需要编译安装findbugs)
下载地址:ant.apache.org/bindownload.cgi
6.CMake 2.6 x86-64 or newer (if compiling native code)(hadoop2.1.0源码中BUILDING.txt中要求使用2.6 or newer)
下载地址:www.cmake.org/cmake/resources/software.html
7.findbugs-2.0.2-source.zip(hadoop2.1.0源码中BUILDING.txt中要求使用1.3.9 or newer)
下载地址:sourceforge.jp/projects/sfnet_findbugs/releases/
如果不安装则报如下错误:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (site) on project hadoop-common: An Ant BuildException has occured: stylesheet /usr/hadoop2.1/hadoop-2.1.0-beta-src/hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn't exist. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-common
8.zlib-devel-1.2.3-3.x86_64.rpm
下载地址:rpm.pbone.net/index.php3/stat/4/idpl/8192688/dir/startcom_5/com/zlib-devel-1.2.3-3.x86_64.rpm.html
如果不安装则报如下错误:
[exec] CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:57 (MESSAGE):
[exec] Could NOT find ZLIB (missing: ZLIB_INCLUDE_DIR)
[exec] Call Stack (most recent call first):
[exec] /usr/share/cmake/Modules/FindZLIB.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
[exec] CMakeLists.txt:89 (find_package)
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-common
9.protobuf-2.5.0.tar.gz
下载地址:code.google.com/p/protobuf/
10.如果全新编译hadoop2.2.0需要打补丁hadoop-10110,怎么做文章最后内容有说明。如果不编译hadoop2.2.0此步骤不需要考虑。
二、准备环境环节中的软件安装
1.安装centos系统,不在这里介绍网上很多教程
2.解压hadoop-2.1.0-beta-src.tar.gz源码到/usr/hadoop/ 命令如下:tar -zxvf hadoop-2.1.0-beta-src.tar.gz -C /usr/hadoop/
3.用rpm命令安装:rpm -ivh jdk-7u40-linux-x64.rpm,如果你安装centos是带有界面的,双击rpm文件即可安装。
安装完成后,配置环境变量,使用vi命令编辑profile文件 vi /etc/profile 移动到最后一行添加以下内容: