hbase 元数据修复

hbase 元数据修复

问题

hbase fck 报错：

..
2021-12-28 16:57:20,906 INFO  [main] util.HBaseFsck: Loading region information from HDFS

2021-12-28 16:57:21,506 INFO  [main] util.HBaseFsck: Checking and fixing region consistency
ERROR: Region { meta => SYSTEM.LOG,\x02\x00\x00\x00,1634711361874.331851809795d177897e9064e961a9f4., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.LOG/331851809795d177897e9064e961a9f4, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => SYSTEM.LOG,\x0B\x00\x00\x00,1634711361874.33d1c4066737ddae24a75456dc080098., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.LOG/33d1c4066737ddae24a75456dc080098, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => SYSTEM.SEQUENCE,,1634711357598.305802e2eaf1eac32b4070a0007dfe85., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.SEQUENCE/305802e2eaf1eac32b4070a0007dfe85, deployed => , replicaId => 0 } not deployed on any region server.2021-12-28 16:57:21,509 INFO  [hbasefsck-pool1-t35] util.HBaseFsck: Region { meta => some_existing_table,,1635386501884.7b8ecde871a1fd5f30c5bd5acd7d44bf., hdfs => hdfs://hacluster/apps/hbase/data/data/default/some_existing_table/7b8ecde871a1fd5f30c5bd5acd7d44bf, deployed => , replicaId => 0 } is in META, and in a disabled tabled that is not deployed

ERROR: Region { meta => SYSTEM.LOG,\x04\x00\x00\x00,1634711361874.b8e44a4251c7321eb9da4dfb969d8d02., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.LOG/b8e44a4251c7321eb9da4dfb969d8d02, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => SYSTEM.LOG,\x14\x00\x00\x00,1634711361874.e3589129a9b0d45bccd3724bac9fa2ed., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.LOG/e3589129a9b0d45bccd3724bac9fa2ed, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => SYSTEM.MUTEX,,1634711364471.37388cfd7c8b0d6bd36e8e134bc6668a., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.MUTEX/37388cfd7c8b0d6bd36e8e134bc6668a, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => SYSTEM.LOG,\x0A\x00\x00\x00,1634711361874.6b8026f87021d59542352fd28f3137ba., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.LOG/6b8026f87021d59542352fd28f3137ba, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => SYSTEM.LOG,\x09\x00\x00\x00,1634711361874.3ddb5dea6d3e4243baa5ab3371574e24., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.LOG/3ddb5dea6d3e4243baa5ab3371574e24, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => SYSTEM.LOG,\x03\x00\x00\x00,1634711361874.c62f1386d7aefd1d8ea1459972e1c949., hdfs => hdfs://hacluster/apps/hbase/data/data/default/SYSTEM.LOG/c62f1386d7aefd1d8ea1459972e1c949, deployed => , replicaId => 0 } not deployed on any region server.
2021-12-28 16:57:21,556 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table SYSTEM.SEQUENCE
ERROR: There is a hole in the region chain between \x02\x00\x00\x00 and \x05\x00\x00\x00.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: There is a hole in the region chain between \x09\x00\x00\x00 and \x0C\x00\x00\x00.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: There is a hole in the region chain between \x14\x00\x00\x00 and \x15\x00\x00\x00.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table SYSTEM.LOG
ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table SYSTEM.MUTEX
Summary:
Table hbase:meta is okay.
    Number of regions: 1
    Deployed on:  pass-bigdata-hadoop-022.chinatelecom.cn,16020,1640095479564
Table hbase:acl is okay.
    Number of regions: 1

解决

有的region上线有问题 用工具给触发上线了

hbase hbck -j /home/hbase-hbck2-1.1.0-last.jar -s assigns c62f1386d7aefd1d8ea1459972e1c949

Hbck异常定位和修复

这有点东西

官方参考