记录一次HBase异常处理过程

一、检查Hbase

使用hbase hbck -details 检查HBase发现多处异常，包括以下四种异常

1、HDFS数据问题

ERROR: Found lingering reference file

2、MetaData问题

ERROR: Region meta listed in hbase:meta on region server aaa but found on region server bbb

ERROR: Region meta => null on HDFS, but not listed in hbase:meta or deployed on any region server

3、regions重叠问题

ERROR: Multiple regions have the same startkey/endkey

ERROR: regions There is an overlap in the region chain.

4、表数据不一致

ERROR: Found inconsistency in table ods_jzgl15:eaj_jz

二、解决异常

问题1解决方法

ERROR: Found lingering reference file hdfs://emr-cluster/hbase/data/ods_jzgl15/eaj_jz/xxx

修复方法：

         $ su - hbase

         $ hbase hbck -fixReferenceFiles eaj_jz

问题2-4解决方法

         ERROR: Region { meta => null, hdfs => hdfs://emr-cluster/hbase/data/ods_jzgl15/eaj_jz/5e4643e6acf8823fbe7bfd1baa9aac9f, deployed => , replicaId => 0 } on HDFS, but not listed in hbase:meta or deployed on any region server

修复方法：

先关闭表：

hbase(main):002:0> disable "ods_jzgl15:eaj_jz"

再执行：

$ hbase hbck -fixMeta -fixAssignments "ods_jzgl15:eaj_jz" > 20210407_1st_tried_repaired_eaj_jz 2>&1 &

$ hbase hbck -fixAssignments "ods_jzgl15:eaj_jz"     > 20210407_2nd_tried_repaired_eaj_jz 2>&1 &

$ hbase hbck

上述命令未生效：

查看修复帮助：

hbase hbck --help

Metadata Repair shortcuts

   -repair           Shortcut for -fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps -fixReferenceFiles -fixTableLocks –fixOrphanedTableZnodes

执行repair操作（该命令为高危操作，谨慎操作）：

$ hbase hbck -repair "ods_jzgl15:eaj_jz" > 20210407_3td_tried_repaired_eaj_jz 2>&1 &

再次执行检查，异常全部修复

$ hbase hbck