Hbase基于 hadoop,若 Hbase 直接使用 release 版本的 hadoop,可能出现数据丢失的情况,hbase 需要使用 hadoop-append ,详细介绍参考Hbase 官网资料
下面以 hbase-0.90.2 为例,介绍 hadoop-0.20.2-append 的编译,以下操作参考:
Building an Hadoop 0.20.x version for HBase 0.90.2
操作系统:CentOS Linux
所需工具:git、ant
[zhankunlin@IctHTC ~]$ mkdir hadoop-append-for-hbase[zhankunlin@IctHTC ~]$ cd hadoop-append-for-hbase
[zhankunlin@IctHTC hadoop-append-for-hbase]$ git clone http://git.apache.org/hadoop-common.git
[zhankunlin@IctHTC hadoop-append-for-hbase]$ cd hadoop-common
[zhankunlin@IctHTC hadoop-common]$ git checkout -t origin/branch-0.20-append
[zhankunlin@IctHTC hadoop-common]$ git show-branch branch-0.20-append
[zhankunlin@IctHTC hadoop-common]$ git show-branch release-0.20.2
[zhankunlin@IctHTC hadoop-common]$ vi ../build.properties
#this is essential
resolvers=internal
#you can increment this number as you see fit
version=0.20-append-for-hbase-zkl
project.version=${version}
hadoop.version=${version}
hadoop-core.version=${version}
hadoop-hdfs.version=${version}
hadoop-mapred.version=${version}
[zhankunlin@IctHTC hadoop-common]$ ln -s ../build.properties build.properties
[zhankunlin@IctHTC hadoop-common]$ pwd
/home/zhankunlin/hadoop-append-for-hbase/hadoop-common
安装 ant
[zhankunlin@IctHTC ~]$ tar zxf apache-ant-1.8.2-bin.tar.gz
[zhankunlin@IctHTC ~]$ export PATH=/home/zhankunlin/apache-ant-1.8.2/bin:$PATH
编译 hadoop
[zhankunlin@IctHTC ~]$ cd hadoop-append-for-hbase/hadoop-common[zhankunlin@IctHTC hadoop-common]$ git checkout branch-0.20-append 确保当前在使用 branch-0.20-append
Already on 'branch-0.20-append'
[zhankunlin@IctHTC hadoop-common]$ ant mvn-install
Buildfile: /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build.xml
.
.
.
mvn-install:
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/hadoop-core-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-core/0.20-append-for-hbase-zkl/hadoop-core-0.20-append-for-hbase-zkl.jar
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/hadoop-test-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-test/0.20-append-for-hbase-zkl/hadoop-test-0.20-append-for-hbase-zkl.jar
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/hadoop-tools-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-tools/0.20-append-for-hbase-zkl/hadoop-tools-0.20-append-for-hbase-zkl.jar
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/hadoop-examples-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-examples/0.20-append-for-hbase-zkl/hadoop-examples-0.20-append-for-hbase-zkl.jar
[artifact:install] [INFO] Installing /home/zhankunlin/hadoop-append-for-hbase/hadoop-common/build/contrib/streaming/hadoop-streaming-0.20-append-for-hbase-zkl.jar to /home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-streaming/0.20-append-for-hbase-zkl/hadoop-streaming-0.20-append-for-hbase-zkl.jar
BUILD SUCCESSFUL
Total time: 11 minutes 42 seconds
可选操作:
[zhankunlin@IctHTC hadoop-common]$ ant test 测试所有功能,花很长时间
[zhankunlin@IctHTC hadoop-common]$ ant test-core 测试核心功能
查看编译好的 jar包(在当前用户主目录下)
[zhankunlin@IctHTC hadoop-common]$ find ~/.m2/repository -name "hadoop-*.jar"
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-tools/0.20-append-for-hbase-zkl/hadoop-tools-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-test/0.20-append-for-hbase-zkl/hadoop-test-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-examples/0.20-append-for-hbase-zkl/hadoop-examples-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-streaming/0.20-append-for-hbase-zkl/hadoop-streaming-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-core/0.20-append-for-hbase-zkl/hadoop-core-0.20-append-for-hbase-zkl.jar
/home/zhankunlin/.m2/repository/org/apache/hadoop/hadoop-core/0.20.2/hadoop-core-0.20.2.jar
编译好的包有:
[zhankunlin@IctHTC hadoop-common]$ find ~/.m2/repository -name "hadoop-*.jar" | awk -F"/" '{print $11}'
hadoop-tools-0.20-append-for-hbase-zkl.jar
hadoop-test-0.20-append-for-hbase-zkl.jar
hadoop-examples-0.20-append-for-hbase-zkl.jar
hadoop-streaming-0.20-append-for-hbase-zkl.jar
hadoop-core-0.20-append-for-hbase-zkl.jar
hadoop-core-0.20.2.jar (这个包不需要)
注意:
这里 0.20-append-for-hbase-zkl 是在build.properties文件中定义的版本好号 version 值,即:
hadoop-tools-VERSION.jar
hadoop-test-VERSION.jar
hadoop-examples-VERSION.jar
hadoop-streaming-VERSION.jar
hadoop-core-VERSION.jar
将编译好的 jar 包复制出来.
[zhankunlin@IctHTC hadoop-common]$ mk ../ok_jars
[zhankunlin@IctHTC hadoop-common]$ jars=`find ~/.m2/repository -name "hadoop-*.jar"`; for jar in $jars; do cp $jar ../ok_jars/ ; done
接下来就是 将编译生成的新jar替换 hadoop-0.20.2-release 和 hbase-0.90.2 中旧的jar包
(1) 替换旧的hadoop包;
请注意,替换 hadoop 中旧的jar包前需要对编译生成的新jar包重新命名。
Hadoop 0.20.2 release 版本中jar包的命名规则为 hadoop-VERSION-PACKAGE.jar, 如:hadoop-0.20.2-examples.jar。
而新编译的jar包的命名规则为 hadoop-PACKAGE-VERSION.jar,如: hadoop-examples-0.20-append-for-hbase.jar.
所以需要将新编译的jar包按如下方式重命名以符合hadoop:hadoop-examples-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-examples.jar
hadoop-test-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-test.jar
hadoop-tools-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-tools.jar
hadoop-streaming-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-streaming.jar
hadoop-core-0.20-append-for-hbase-zkl.jar --> hadoop-0.20-append-for-hbase-zkl-core.jar
(2) 替换hbase中lib文件夹中的包
Hbase使用的命名规则为hadoop-PACKAGE-VERSION.jar,和新编译的jar包命名规则一致,所以只需要直接复制jar包到$HBASE_HOME/lib目录下,不需要重命名。