[hadoop、Hbase错误解决合集] Hbase的Hmaster进程启动后自动关闭的问题解决方法

==============================================================================

现象
1.日志中提示：
ERROR [Thread-14] master.HMaster: Failed to become active master
2.进入hbase shell中进行数据库操作提示：
KeeperErrorCode = NoNode for /hbase/master

解决步骤：

一、首先打开hbase的日志文件，一般默认是在hbase的logs文件夹中

2019-04-01 12:25:37,742 ERROR [Thread-14] master.HMaster: Failed to become active master
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1083)
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:421)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:611)
	at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1407)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:853)
	at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2241)
	at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:567)
	at java.lang.Thread.run(Thread.java:748)
2019-04-01 12:25:37,743 ERROR [Thread-14] master.HMaster: ***** ABORTING master localhost,16000,1554092730588: Unhandled exception. Starting shutdown. *****
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1083)
	at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:421)
	at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:611)
	at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1407)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:853)
	at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2241)
	at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:567)
	at java.lang.Thread.run(Thread.java:748)

二、通过分析可知， ERROR [Thread-14] master.HMaster: Failed to become active master是导致Hmaster进程关闭的原因。

三、查看错误描述
java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of ‘hbase.procedure.store.wal.use.hsync’ to set the desired level of robustness and ensure the config value of ‘hbase.wal.dir’ points to a FileSystem mount that can provide it.

程序WAL（预写日志）依赖于hsync（行同步信号）能力在组件故障期间进行正确操作，但是底层文件系统不支持这样做。请检查’hbase.procedure.store.wal.use '的配置值。设置所需的健壮性级别，并确保配置值为’hbase.wal.dir’指向可以提供它的文件系统挂载。

---

关于hbase中hsync的简单介绍
hsync的语义是：client端所有的数据都发送到副本的每个datanode上，并且datanode上的每个副本都完成了posix中fsync的调用，也就是说操作系统已经把数据刷到磁盘上（当然磁盘也可能缓冲数据）；需要注意的是当调用fsync时只有当前的block会刷到磁盘中，要想每个block都刷到磁盘，必须在创建流时传入Sync标示。
通过分析上面的语义，可以确定hsyn涉及到两个角色client和DataNode

四、再看hbase中关于stream capabilities（hflush/hsync）功能的建议。

Controls whether HBase will check for stream capabilities (hflush/hsync).

          Disable this if you intend to run on LocalFileSystem, denoted by a rootdir
          with the 'file://' scheme, but be mindful of the NOTE below.
（如果你打算在本地文件系统上运行，请禁用此选项。）
          WARNING: Setting this to false blinds you to potential data loss and
          inconsistent system state in the event of process and/or node failures. If
          HBase is complaining of an inability to use hsync or hflush it's most
          likely not a false positive.
   （将此设置为false会使您在节点故障时看不到潜在的数据丢失和进程中不一致的系统状态。如果HBase提示无法使用hsync或hflush，那么它很可能不是一个误报。）

---

在我的机器上，hadoop、hbase都是配置的伪分布式，DataNode只有一个，hbase运行在LocalFileSystem（由rootdir表示），所以在配置文件（vim /usr/local/hbase/conf/hbase-site.xml）中关闭stream capabilities（hflush/hsync）功能即可。

具体要添加的配置内容如下

<property>
  <name>hbase.unsafe.stream.capability.enforce</name>
  <value>false</value>
</property>

五、查看结果

[hadoop@localhost ~]$ start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [localhost.localdomain]
[hadoop@localhost ~]$ start-hbase.sh
localhost: running zookeeper, logging to /usr/local/hbase/logs/hbase-hadoop-zookeeper-localhost.localdomain.out
running master, logging to /usr/local/hbase/logs/hbase-hadoop-master-localhost.localdomain.out
: running regionserver, logging to /usr/local/hbase/logs/hbase-hadoop-regionserver-localhost.localdomain.out
[hadoop@localhost ~]$ jps
17682 HMaster
17091 SecondaryNameNode
16708 NameNode
17847 HRegionServer
18316 Jps
16845 DataNode
17533 HQuorumPeer

[hadoop@localhost ~]$ hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.0.5, r76458dd074df17520ad451ded198cd832138e929, Mon Mar 18 00:41:49 UTC 2019
Took 0.0060 seconds                                                             
hbase(main):001:0> create 'test','cf'
Created table test
Took 1.8818 seconds                                                             
=> Hbase::Table - test
hbase(main):002:0> exit

一切正常，问题解决！！！

=======================================================================
  

参考资料：
问题的解决方法来自这里。
https://blog.csdn.net/SL_World/article/details/84952725#commentsedit
HDFS中hsync方法介绍
https://www.cnblogs.com/yangjiandan/p/3540498.html
hadoop官方对stream capabilities的定义
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/StreamCapabilities.html

=======================================================================