一次region过多导致HBase服务宕机事件

         具体情况是，甲方有10个节点的HBase集群，主要业务表共10张，region总数达23000+，平均每台RegionServer（RS）的region数量2300左右，每台RS堆内存配置96G（初始配置32G，节点内存256G）。在业务正常运行几个月后，所有RS突然宕机。

问题分析

         因为集群region数量太多了，导致每个region的memstore空间缩小，memstore不断执行flush操作，也可理解为节点所有region的memstore总大小不断达到global memstore大小，所以不断触发flush机制。

          因此，在实时数据高频写入时，频繁的flush会产生了很多hfile文件，从宕机前日志看到memstore不到10M就会flush一次，hfile文件过多导致minor compaction持续进行。不断的flush与compaction给HDFS造成了很大压力，导致DataNode负载过高，最终RegionServer写HDFS异常，RegionServer全部宕机。

相关日志

RegionServer宕机日志： 

2018-12-19 07:59:50,174 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
java.io.IOException: Unable to create new block.
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1633)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:772)
2018-12-19 07:59:50,174 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/hbase/data/db1/xxx_position/d9fa4a857e668b63db02a5e3340354de/recovered.edits/0000000000023724243.temp" - Aborting...
2018-12-19 07:59:50,175 WARN org.apache.hadoop.hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[172.xx.xx.xx:50010,DS-bae224ff-1680-4969-afd1-1eeae878783b,DISK]
2018-12-19 07:59:50,177 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 172.xx.xx.xx:50010
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1722)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1620)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:772)

···

2018-12-19 07:59:50,610 INFO org.apache.hadoop.hbase.wal.WALSplitter: 3 split writers finished; closing...
2018-12-19 07:59:50,613 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.io.EOFException: Premature EOF: no length prefix available
        at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1701)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1620)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:772)
 

DataNode日志：

2018-12-19 08:16:07,582 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1452924002-172.16.10.18-1527602017139:blk_1181831924_108091843, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.EOFException: Premature EOF: no length prefix available
        at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1286)
        at java.lang.Thread.run(Thread.java:748)
2018-12-19 08:16:07,582 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-1452924002-172.16.10.18-1527602017139:blk_1181831924_108091843
java.io.IOException: Premature EOF from inputStream
        at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:500)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:896)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:808)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
        at java.lang.Thread.run(Thread.java:748)
2018-12-19 08:16:07,582 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.run():
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.DataOutputStream.flush(DataOutputStream.java:123)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1533)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1470)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1383)
        at java.lang.Thread.run(Thread.java:748)

···
    
2018-12-19 22:08:15,622 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: hadoop-data-10.cloud:50010:DataXceiverServer:
java.io.IOException: Xceiver count 4097 exceeds the limit of concurrent xcievers: 4096
        at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:149)
        at java.lang.Thread.run(Thread.java:748)
2018-12-19 22:08:15,622 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: hadoop-data-10.cloud:50010:DataXceiverServer:
java.io.IOException: Xceiver count 4098 exceeds the limit of concurrent xcievers: 4096
        at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:149)
        at java.lang.Thread.run(Thread.java:748) 

处理过程

       将DataNode最大传输线程数dfs.datanode.max.transfer.threads从默认4096调整到32768，经过一段时间hdfs缓慢恢复。由于hdfs数据块过多，所以导致hdfs恢复耗时比较久。

      然后，重启HBase，又遇到主Master一直启动失败。查看Master日志：

2018-12-21 13:11:51,629 ERROR org.apache.hadoop.hbase.master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.
2018-12-21 16:15:20,459 FATAL org.apache.hadoop.hbase.master.HMaster Failed to become active master
java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned
    at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:106)
    at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1057)
    at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:844)
    at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:194)
    at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1834)
    at java.lang.Thread.run(Thread.java:748)
2018-12-21 16:15:20,472 FATAL org.apache.hadoop.hbase.master.HMaster Master server abort: loaded coprocessors are: []
2018-12-21 16:15:20,472 FATAL org.apache.hadoop.hbase.master.HMaster Unhandled exception. Starting shutdown.
java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned
    at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:106)
    at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1057)
    at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:844)
    at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:194)
    at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1834)
    at java.lang.Thread.run(Thread.java:748) 

        分析Master一直启动失败的原因是region太多导致HMaster初始化超时（默认900000，15分钟）；加上HMaster初始化时并不会优先加载hbase:namespace表，默认5分钟如果加载不到namespace表就会超时，同样会造成HMaster初始化失败。

        随后参考HBase源码，调整了初始化超时时间等参数，最终启动成功，以下是具体调整参数

<!-- namespace系统表assign超时时间，默认300000，因为region太多这里设置1天保证初始化成功 -->
<name>hbase.master.namespace.init.timeout</name> 
<value>86400000</value>

<!-- master初始化超时时间，默认900000，因为region太多这里设置1天保证初始化成功 -->
<name>hbase.master.initializationmonitor.timeout</name> 
<value>86400000</value>

<!-- bulk assign region超时时间，默认300000，设置1小时充分保证每个bulk assign都能成功 -->
<name>hbase.bulk.assignment.waiton.empty.rit</name> 
<value>3600000</value>

<!-- bulk assign每个region打开的时间，默认1000，这里也尽量设置大些比如30s -->
<name>hbase.bulk.assignment.perregion.open.time</name> 
<value>30000</value>

总结

        本次HBase RS宕机是因为Unable to create new block即创建不了数据块，分析下来是集群规模太小，region数量过多，实时数据高频写入不断flush与compaction，造成HDFS上小文件过多，DataNode创建文件失败，从而导致RegionServer整体宕机。

        处理过程中又遇到HMaster初始化超时、namespace表加载超时，调整默认启动参数后才正常启动。此次事故也让我们明白了小集群过载时会带来什么风险。