1.工具
----------------------------------------------------------------------------------
Requirements:
* Unix System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
----------------------------------------------------------------------------------
安装jdk
http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.tar.gz?AuthParam=1395828777_fcd88f7a3b2c6b67e39c64adec67acc3
安装maven
http://apache.dataguru.cn/maven/maven-3/3.1.1/source/apache-maven-3.1.1-src.tar.gz
安装Findbugs
http://ncu.dl.sourceforge.net/project/findbugs/findbugs/2.0.3/findbugs-2.0.3.tar.gz
安装protobuf
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2
hadoop2.3.0源码
http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.3.0/hadoop-2.3.0-src.tar.gz
参考:http://wiki.apache.org/hadoop/HowToContribute
2.解压源码,然后用mvn编译mvn package -Pdist,native,docs -DskipTests -Dtar
二、Hadoop 2.3.0集群配置
配置项参考:http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html
安装包下载地址:http://apache.fayea.com/apache-mirror/hadoop/common/hadoop-2.3.0/hadoop-2.3.0.tar.gz
1.jdk配置
2.ssh无密码登陆
3.配置hdfs-site.xml ,core-site.xml,mapred-site.xml,yarn-site.xml,slaves,yarn-env.sh,haoop-env.sh(jdk位置)
4.启动hadoop start-dfs.sh start-yarn.sh
附:
hadoop编译参考:
http://www.micmiu.com/bigdata/hadoop/hadoop-build-source-2-2-0/
http://blog.chinaunix.net/uid-9460004-id-4047994.html
hadoop子项目配置参考:
http://m.blog.csdn.net/blog/jinnchang/8900992
http://wenku.baidu.com/view/fe1b2f22de80d4d8d15a4f6e.html
注:11/09/20 17:29:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 出现这样的错误
1.指定本地库的路径,hadoop-env.sh
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
2.库不兼容本机机器系统,本地maven编译hadoop项目,将编译所得的本地库替换libhadoop.so就可以了。
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html