一. 问题描述
发现hbase集群中有一个表的region在SPLITING 状态持续很久不结束,HMaster节点进行full gc 可以回收 。
同时在此期间提交的创建表和drop表的操作无效,显示在ENABLING和DISABLING状态,无法成功完成创建表和删除表的操作
test_mgq ENABLING
test_userfriend DISABLING
test_userfriendlwj ENABLING
在HMaster报错信息如下
2014-04-09 00:17:52,007 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: mau_selecteduser,2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF1QQQQQQQQQ,1396415717740.50aa699775f00b753aaaa6e029fed887. state=SPLITTING, ts=1396415877142, server=YZ18-134.opi.com,60020,1395937706714
2014-04-09 00:17:59,706 INFO org.apache.hadoop.hbase.master.handler.DisableTableHandler: Offlining 1 regions.
2014-04-09 00:18:02,007 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: mau_selecteduser,2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF1QQQQQQQQQ,1396415717740.50aa699775f00b753aaaa6e029fed887. state=SPLITTING, ts=1396415877142, server=YZ18-134.opi.com,60020,1395937706714
2014-04-09 00:18:12,007 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: mau_selecteduser,2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF1QQQQQQQQQ,1396415717740.50aa699775f00b753aaaa6e029fed887. state=SPLITTING, ts=1396415877142, server=YZ18-134.opi.com,60020,1395937706714
2014-04-09 00:18:22,007 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: mau_selecteduser,2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF1QQQQQQQQQ,1396415717740.50aa699775f00b753aaaa6e029fed887. state=SPLITTING, ts=1396415877142, server=YZ18-134.opi.com,60020,1395937706714
2014-04-09 00:18:32,007 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: mau_selecteduser,2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF1QQQQQQQQQ,1396415717740.50aa699775f00b753aaaa6e029fed887. state=SPLITTING, ts=1396415877142, server=YZ18-134.opi.com,60020,1395937706714
2014-04-09 00:18:39,602 DEBUG org.apache.hadoop.hbase.client.ClientScanner: Creating scanner over .META. starting at key ''
2014-04-09 00:18:39,602 DEBUG org.apache.hadoop.hbase.client.ClientScanner: Advancing internal scanner to startKey at ''
2014-04-09 00:18:40,220 DEBUG org.apache.hadoop.hbase.client.ClientScanner: Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
2014-04-09 00:18:41,549 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 16924 catalog row(s) and gc'd 0 unreferenced parent region(s)
2014-04-09 00:18:42,007 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: mau_selecteduser,2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF1QQQQQQQQQ,1396415717740.50aa699775f00b753aaaa6e029fed887. state=SPLITTING, ts=1396415877142, server=YZ-134.opi.com,60020,1395937706714
2014-04-09 00:18:52,007 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: mau_selecteduser,2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF1QQQQQQQQQ,1396415717740.50aa699775f00b753aaaa6e029fed887. state=SPLITTING, ts=1396415877142, server=YZ-134.opi.com,60020,1395937706714
2014-04-09 00:19:02,007 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: mau_selecteduser,2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF2\xF1QQQQQQQQQ,1396415717740.50aa699775f00b753aaaa6e029fed887. state=SPLITTING, ts=1396415877142, server=YZ18-134.opi.com,60020,1395937706714
2014-