1.一次Hbase Region In Trasition问题的解决过程

最新推荐文章于 2021-09-26 17:09:40 发布

终回首

最新推荐文章于 2021-09-26 17:09:40 发布

阅读量484

点赞数

分类专栏： # Apache Hbase 文章标签： hbase

本文链接：https://blog.csdn.net/qq_39945938/article/details/103752706

版权

Apache Hbase 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

一次Hbase Region In Trasition问题的解决过程

一、问题

1.打开hbase ui界面，提示有一个region进入Region In Trasition 状态。

2.执行hbase hbck 命令

日志：

19/12/06 19:15:06 INFO util.HBaseFsck: Validating mapping using HDFS state
Number of live region servers: 4
Number of dead region servers: 0
Master: bigdata001,16000,1575629127668
Number of backup masters: 0
Average load: 100.5
Number of requests: 594314
Number of regions: 402
Number of regions in transition: 1
19/12/06 19:15:06 INFO util.HBaseFsck: Loading regionsinfo from the hbase:meta table


Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
19/12/06 19:15:06 INFO util.HBaseFsck: getTableDescriptors == tableNames => [KYLIN_S7M90IVW4Y, KPI_ENVENT, KYLIN_P16OISRJPG, KYLIN_5DDEZJX89A, KYLIN_4E7AJM0RL2, SYSTEM:STATS, KYLIN_64FD9B1SKK, KYLIN_SKGJAK2Q1I,
KYLIN_4TH0QZNKY0, KYLIN_PJC35CMSLT, kylin_metadata, KYLIN_XVKUJNAOKK, SYSTEM:MUTEX, QARresult_filted1, KYLIN_GEDSW452EJ, KYLIN_AFXNCN44PA, QARresult_filted_HP, KPI_ENVENT_HP, KYLIN_Z0VZWO6Y0Q, KPI_ENVENT_TEST1, KYLIN_HFG9S1KV5Q, KYLIN_0LKP7VMOIR, KYLIN_LPH23EKTU5, KYLIN_NNUS4SPU72, KYLIN_FWUB0K941D, KYLIN_9ID2WQ81WB, SYSTEM:SEQUENCE, KYLIN_69NWJ4AGRX, KPI_ENVENT_V2_copy2, KYLIN_WWRZZMJRHD, KYLIN_HPWK085GCA, hbase:namespace, KYLIN_BJ8LVD1OX4, qarresulttest1, KYLIN_7WXPX506DI, SYSTEM:FUNCTION, KYLIN_NRTO2AFU4A, KYLIN_5M0ZSS7WQ9, direct_fly, qarresulttest_HP, for_test1, ss, KYLIN_P6KOHWO47O, KYLIN_7DH8MZMH9B, acars, KPI_ENVENT_V2, qarresult_zl, qarfilted_zl, KYLIN_GNX424TEDP, KYLIN_XE8JVMFR4D, flight_route, KYLIN_YJKYIY2RHJ, KYLIN_AH8PJ46DYM, SYSTEM:LOG, KYLIN_6V8P23KVB1, KYLIN_6ODTP8WV1C, SYSTEM:CATALOG, KPI_ENVENT_V1, QARresult_filted2]19/12/06 19:15:06 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=bigdata002:2181,bigdata003:2181,bigdata001:2181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZK
Client$$Lambda$13/1891236557@35bb7abe19/12/06 19:15:06 INFO zookeeper.ClientCnxn: Opening socket connection to server bigdata002/172.17.100.239:2181. Will not attempt to authenticate using SASL (unknown error)
19/12/06 19:15:06 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.17.100.243:54662, server: bigdata002/172.17.100.239:2181
19/12/06 19:15:06 INFO zookeeper.ClientCnxn: Session establishment complete on server bigdata002/172.17.100.239:2181, sessionid = 0x16ea068c9ac92c1, negotiated timeout = 120000
19/12/06 19:15:06 INFO client.ConnectionImplementation: Closing master protocol: MasterService
Number of Tables: 59
19/12/06 19:15:06 INFO zookeeper.ZooKeeper: Session: 0x16ea068c9ac92c1 closed
19/12/06 19:15:06 INFO zookeeper.ClientCnxn: EventThread shut down
19/12/06 19:15:06 INFO util.HBaseFsck: Loading region directories from HDFS
.............
19/12/06 19:15:06 INFO util.HBaseFsck: Loading region information from HDFS
....
19/12/06 19:15:06 INFO util.HBaseFsck: Checking and fixing region consistency
19/12/06 19:15:06 INFO util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.

二、解决

排查过程：

1.切换到hdfs,执行hbase hbck 查看hbase状态

2.根据Error提示，删除对应的region的元数据和数据（这里直接删除是因为region所属的表已经被删除了）。

deleteall 'table','rowkey'

3.出现问题的原因：

hbase表的hdfs文件和元数据表不一致，可能是文件丢失了，也可能是hbase:meta表数据没有region的信息了。此时，删除表和对应region的hdfs文件和数据文件，删除hbase:meta表的对应记录即可
hbase表删除成功，但hbase在zookeeper的节点还存在，此时要递归删除zookeeper 的/hbase节点，然后重启zookeeper和hbase即可