HBase 2.2.2 on Hadoop 3.2.1源码编译

在《HBase 2.2.2 on Hadoop 3.2.1完全分布式部署》中，我们提到需要在hbase-site.xml中增加如下配置：

<property> <name>hbase.unsafe.stream.capability.enforce</name> <value>false</value> <description> Controls whether HBase will check for stream capabilities (hflush/hsync). Disable this if you intend to run on LocalFileSystem, denoted by a rootdir with the 'file://' scheme, but be mindful of the NOTE below. WARNING: Setting this to false blinds you to potential data loss and inconsistent system state in the event of process and/or node failures. If HBase is complaining of an inability to use hsync or hflush it's most likely not a false positive. </description> </property>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

<property>     <name>hbase.unsafe.stream.capability.enforce</name>     <value>false</value>     <description>       Controls whether HBase will check for stream capabilities (hflush/hsync).         Disable this if you intend to run on LocalFileSystem, denoted by a rootdir       with the 'file://' scheme, but be mindful of the NOTE below.         WARNING: Setting this to false blinds you to potential data loss and       inconsistent system state in the event of process and/or node failures. If       HBase is complaining of an inability to use hsync or hflush it's most       likely not a false positive.     </description> </property>

才能解决由于文件系统不支持hsync报错而造成启动失败的问题：

2019-12-03 21:37:46,032 ERROR [master/hadoop-master:16000:becomeActiveMaster] master.HMaster: Failed to become active master java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it. at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1092) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:424) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:576) at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1528) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:938) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2112) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:580) at java.lang.Thread.run(Thread.java:748) 2019-12-03 21:37:46,032 ERROR [master/hadoop-master:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master hadoop-master,16000,1575380253796: Unhandled exception. Starting shutdown. *****

1 2 3 4 5 6 7 8 9 10 11

2019-12-03 21:37:46,032 ERROR [master/hadoop-master:16000:becomeActiveMaster] master.HMaster: Failed to become active master java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.         at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1092)         at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:424)         at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:576)         at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1528)         at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:938)         at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2112)         at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:580)         at java.lang.Thread.run(Thread.java:748) 2019-12-03 21:37:46,032 ERROR [master/hadoop-master:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master hadoop-master,16000,1575380253796: Unhandled exception. Starting shutdown. *****

但至于为何需要这个的配置大多都语焉不详，而且官网明确指出了该参数仅适用于本地文件系统：

Master fails to become active due to lack of hsync for filesystem

结合https://stackoverflow.com/questions/48709569/hbase-error-illegalstateexception-when-starting-master-hsync中

HBase 2.2.2二进制发行版中使用的hadoop 2.x的依赖

的说法，查看hbase/lib目录中确实是使用的hadoop 2.8.5的库：

HBase 2.2.2二进制发行版中默认使用的是Hadoop 2.x的依赖（2.8.5）

决定编译源码进行尝试。否则根据官网的说法，hbase.unsafe.stream.capability.enforce=false存在丢失数据的风险，根本无法用于生产环境。

从官网下载HBase2.2.2源码hbase-2.2.2-src.tar.gz，使用tar zvxf hbase-2.2.2-src.tar.gz解压，然后切换到hbase-2.2.0目录下（有pom.xml）直接使用maven打包：

mvn clean package -DskipTests assembly:single -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1

1	mvn clean package -DskipTests assembly:single -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1

根据官网http://hbase.apache.org/book.html中的《Building Apache HBase》一节：

HBase supports building against Apache Hadoop versions: 2.y and 3.y (early release artifacts). By default we build against Hadoop 2.x.

因而可以使用上述的命令“Building against various hadoop versions”。如果从来没有下载过依赖的话，不使用maven的mirror大约需要一个小时以上，主要耗时都在各种下载依赖，耐心等待直到打包完成：

HBase 2.2.2 on Hadoop 3.2.1源码编译

编译后的结果位于hbase-assembly/target文件夹下：

root@iZ2ze66x0wy26jjx3ibxwoZ:~/hbase/hbase-2.2.2/hbase-assembly/target# pwd /root/hbase/hbase-2.2.2/hbase-assembly/target root@iZ2ze66x0wy26jjx3ibxwoZ:~/hbase/hbase-2.2.2/hbase-assembly/target# ls archive-tmp dependency dependency-maven-plugin-markers hbase-2.2.2-bin.tar.gz hbase-2.2.2-client-bin.tar.gz maven-shared-archive-resources NOTICE.aggregate supplemental-models.xml root@iZ2ze66x0wy26jjx3ibxwoZ:~/hbase/hbase-2.2.2/hbase-assembly/target#

1 2 3 4 5

root@iZ2ze66x0wy26jjx3ibxwoZ:~/hbase/hbase-2.2.2/hbase-assembly/target# pwd /root/hbase/hbase-2.2.2/hbase-assembly/target root@iZ2ze66x0wy26jjx3ibxwoZ:~/hbase/hbase-2.2.2/hbase-assembly/target# ls archive-tmp  dependency  dependency-maven-plugin-markers  hbase-2.2.2-bin.tar.gz  hbase-2.2.2-client-bin.tar.gz  maven-shared-archive-resources  NOTICE.aggregate  supplemental-models.xml root@iZ2ze66x0wy26jjx3ibxwoZ:~/hbase/hbase-2.2.2/hbase-assembly/target#

可以看到其中的hbase-2.2.2-bin.tar.gz，将其解压，观察lib中的已经改为依赖于3.2.1。此时去掉hbase.unsafe.stream.capability.enforce=false的配置后也可以正常启动了HMaster了。

Windows环境下可以使用cygwin环境解压tar包并编译，需要注意源码tar包的解压是否正常，不要使用cmd.exe。

参考文档：
1、http://hbase.apache.org/book.html
2、https://blog.csdn.net/ccren/article/details/93923414