HBase Region数据不一致

【背景】

应用在进行bulkload将hfile导入hbase中报错：

2019-04-03 11:27:18,509 [LoadIncrementalHFiles-2][org.apache.hadoop.hbase.client.RpcRetryingCaller:132] [INFO ] - Callexception, tries=20, retries=35, started=269051 ms ago, cancelled=false,msg=row '048000-1229171819-48889202-48889202-0-79818770#090' on table'tbl_glhis_hb_swt_addn_inf' atregion=tbl_glhis_hb_swt_addn_inf,048000-1229171819-48889202-48889202-0-79818770#090,1554144453180.10bd9b9ffb8bcaac4f939a79a29f4ba1.,hostname=y3050705,60020,1517293052636, seqNum=1

查看hbase master web ui页面发现tbl_glhis_hb_swt_addn_inf在y3050705节点上的region不可用，手工停止该节点上的regionserver服务。应用再次发起，报错依然。

【修复过程】

任意在一台regionserver节点上执行

export HADOOP_USER_NAME=hbase

hbase hbck -details

tbl_glhis_hb_swt_addn_inf > 1.txt 2>&1

查看1.txt日志报错如下：

---- Table 'tbl_achis_hb_trans_flow': overlap groups

There are 0 overlap groups with 0 overlapping regions

19/04/03 13:40:15 INFO util.HBaseFsck: Computingmapping of all store files

...........................................................................................java.lang.OutOfMemoryError:Java heap space

Dumping heap to java_pid16636.hprof ...

Heap dump file created [330952255 bytes in 2.844 secs]

#

# java.lang.OutOfMemoryError: Java heap space

# -XX:OnOutOfMemoryError=