hbase表修复

最新推荐文章于 2024-05-20 13:41:51 发布

时空无限

最新推荐文章于 2024-05-20 13:41:51 发布

阅读量5.3k

点赞数 1

分类专栏： Hadoop生态文章标签： hadoop hbase

本文链接：https://blog.csdn.net/weixin_40548182/article/details/109048734

版权

Hadoop生态专栏收录该内容

2 篇文章 0 订阅

订阅专栏

场景

opentsdb启动报错

grep -i error /tmp/opentsdb.log
2020-10-13 12:19:41,098 INFO  [main-SendThread(localhost:2181)] ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-10-13 12:19:44,717 INFO  [main-SendThread(localhost:2181)] ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-10-13 12:19:48,400 INFO  [main-SendThread(localhost:2181)] ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

cat /tmp/opentsdb.log
省略。。。
2020-10-13 12:21:31,740 WARN  [AsyncHBase I/O Worker #1] HBaseClient: Probe Exists(table="tsdb-uid", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1542978859652.665eaacf411c9f82e13e35a62cfff831.", stop_key="")) failed
org.hbase.async.NonRecoverableException: Too many attempts: Exists(table="tsdb-uid", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1542978859652.665eaacf411c9f82e13e35a62cfff831.", stop_key=""))
	at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) [asynchbase-1.7.2.jar:na]
	at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) [asynchbase-1.7.2.jar:na]
	at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) [asynchbase-1.7.2.jar:na]
	at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) [asynchbase-1.7.2.jar:na]
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na]
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na]
	at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712) [asynchbase-1.7.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:1533) [asynchbase-1.7.2.jar:na]
	at org.hbase.async.RegionClient.decode(RegionClient.java:88) [asynchbase-1.7.2.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]
	at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1223) [asynchbase-1.7.2.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]
	at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3121) [asynchbase-1.7.2.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2020-10-13 12:21:31,743 INFO  [AsyncHBase I/O Worker #2] HBaseClient: Invalidated cache for RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1542978859652.665eaacf411c9f82e13e35a62cfff831.", stop_key="") as RegionClient@477578960(chan=[id: 0xc92eaecf, /172.18.0.240:33258 => /172.18.0.241:16020], #pending_rpcs=0, #batched=0, #rpcs_inflight=0) seems to be splitting or closing it.
2020-10-13 12:21:31,773 INFO  [main] TSDB: Flushing compaction queue
2020-10-13 12:21:32,023 INFO  [main] HBaseClient: Channel [id: 0x3e6bc4b6, /172.18.0.240:57726 => /172.18.0.242:16020] is disconnecting: [id: 0x3e6bc4b6, /172.18.0.240:57726 => /172.18.0.242:16020] DISCONNECT
2020-10-13 12:21:32,024 INFO  [main] HBaseClient: Lost connection with the .META. region
2020-10-13 12:21:32,027 INFO  [main] HBaseClient: Channel [id: 0xc92eaecf, /172.18.0.240:33258 => /172.18.0.241:16020] is disconnecting: [id: 0xc92eaecf, /172.18.0.240:33258 => /172.18.0.241:16020] DISCONNECT
2020-10-13 12:21:32,033 INFO  [main] TSDB: Completed shutting down the TSDB
Exception in thread "main" java.lang.RuntimeException: Initialization failed
	at net.opentsdb.tools.TSDMain.main(TSDMain.java:237)
Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception:
	at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169)
	at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142)
	at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36)
	at com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82)
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
	at com.stumbleupon.async.Deferred.access$300(Deferred.java:430)
	at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
	at com.stumbleupon.async.Deferred.access$300(Deferred.java:430)
	at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366)
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278)
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257)
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1005)
	at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712)
	at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2058)
	at org.hbase.async.HBaseClient.handleNSRE(HBaseClient.java:2848)
	at org.hbase.async.RegionClient.decode(RegionClient.java:1527)
	at org.hbase.async.RegionClient.decode(RegionClient.java:88)
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1223)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)
	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
	at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3121)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.hbase.async.RegionOpeningException: org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region tsdb,,1548705607033.6fb5105995b1d6d49e3bfb5cbef4c1ab. is opening on c7002,16020,1602561238594
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1132)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2075)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

Caused by RPC: Exists(table="tsdb", key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table="tsdb", region_name="tsdb,,1548705607033.6fb5105995b1d6d49e3bfb5cbef4c1ab.", stop_key=[0, 0, 1, 92, 44, 27, 32, 0, 0, 1, 0, 0, 1, 0, 0, 2, 0, 0, 33, 0, 0, 3, 0, 0, 16, 0, 0, 4, 0, 0, 106]))
	at org.hbase.async.RegionOpeningException.make(RegionOpeningException.java:73)
	at org.hbase.async.RegionOpeningException.make(RegionOpeningException.java:34)
	at org.hbase.async.RegionClient.makeException(RegionClient.java:1753)
	at org.hbase.async.RegionClient.decodeException(RegionClient.java:1773)
	at org.hbase.async.RegionClient.decode(RegionClient.java:1485)
	... 29 more

第1章 hbase报错

1、hbase hbck检查报错

hbase hbck
省略。。。
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
省略。。。

hbase修复表

hbase hbck -fixEmptyMetaCells
省略。。。

2、hbase hbck检查报错

hbase hbck
省略。。。
ERROR: There is a hole in the region chain between \x00\x00l^Gl\xC0\x00\x00\x01\x00\x00&\x00\x00\x02\x00\x047\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x04+ and \x00\x00m]\x0C,\x10\x00\x00\x01\x00\x0BC\x00\x00\x02\x00\x03\xCA\x00\x00\x03\x00\x03\xC9\x00\x00\x04\x00\x0BF.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Last region should end with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole.
ERROR: Found inconsistency in table tsdb
省略。。。

这种情况是在hdfs层面上的，这个region的.regioninfo（meta）文件不存在，使用"-fixHdfsHole"进行修复；-fixHdfsHole：修复region holes（空洞，某个区间没有region）问题。
hbase修复表

hbase hbck -fixHdfsHoles
省略。。。

3、hbase hbck检查报错

No HDFS region dir found:
Orphan region in HDFS: Unable to load .regioninfo from table tsdb in hdfs dir hdfs://bdcluster/hbase/data/default/tsdb/4021d8fe56d2d5db82151ac777474c17!  It may be an invali    d format or version file.  Treating as an orphaned regiondir.
省略。。。
Failed to read .regioninfo file for region tsdb
File does not exist: /hbase/data/default/tsdb/4021d8fe56d2d5db82151ac777474c17/.regioninfo

hbase修复表

flush表
[root@c7001 ~]# hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/disk/hbase-1.3.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/disk/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.3.0, re359c76e8d9fd0d67396456f92bcbad9ecd7a710, Tue Jan  3 05:31:38 MSK 2017

hbase(main):001:0> flush tsdb
NameError: undefined local variable or method `tsdb' for #<Object:0x7df28f1>
注意：表名加上引号
hbase(main):007:0> flush 'tsdb'
0 row(s) in 13.6890 seconds
hbase(main):008:0> quit

hbase hbck -fixHdfsOrphans tsdb

4、hbase hbck检查报错

not deployed on any region server.

这种情况下，region的hfile等数据都在，只是没有在任何region上online，使用"fixAssignments"进行修复。-fixAssignments：修复没有assign、assign不正确或者同时assign到多台RegionServer的问题region。

hbase修复表

hbase hbck -fixAssignments

5、hbase hbck检查报错

ERROR: Region { meta => null, hdfs => hdfs://bdcluster/hbase/data/default/tsdb/0ca3a2bf4cf091e97dbe26624c9757e6, deployed => , replicaId => 0 } on HDFS, but not listed in hbase:meta or deployed on any region server

这种情况下region的实际数据是存在的，但是在hbase:meta中不存在，使用"-fixMeta"进行信息同步修复。
hbase修复表

hbase hbck -fixMeta

6、hbase hbck检查报错

is a split parent in META, in HDFS, and not deployed on any region server. This could be transient.

7、hbase hbck检查报错

ERROR: (region tsdb,\x00\x00*]\x98\x15@\x00\x00\x01\x00\x06\x0A\x00\x00\x02\x00\x04\x06\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x06!,1602579999051.9a16ad6816a73c4f9f89ebe66eded8d6.) Multiple regions have the same startkey: \x00\x00*]\x98\x15@\x00\x00\x01\x00\x06\x0A\x00\x00\x02\x00\x04\x06\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x06!
ERROR: (region tsdb,\x00\x00*]\x98\x15@\x00\x00\x01\x00\x06\x0A\x00\x00\x02\x00\x04\x06\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x06!,1589371271384.a52c9d2cdb367b250cb251666d9007b4.) Multiple regions have the same startkey: \x00\x00*]\x98\x15@\x00\x00\x01\x00\x06\x0A\x00\x00\x02\x00\x04\x06\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x06!
ERROR: (regions tsdb,\x00\x00*]\x98\x15@\x00\x00\x01\x00\x06\x0A\x00\x00\x02\x00\x04\x06\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x06!,1589371271384.a52c9d2cdb367b250cb251666d9007b4. and tsdb,\x00\x00*]\xAET\x90\x00\x00\x01\x00\x06\x0A\x00\x00\x02\x00\x00\x06\x00\x00\x03\x00\x00\x03\x00\x00\x04\x00\x06\x1C,1602579999051.b2fe5dec4ad0104a83da059d18cc7da3.) There is an overlap in the region chain.
2020-10-13 17:10:50,249 WARN  [main] util.HBaseFsck: reached end of problem group: \x00\x00*]\xC2\xFC\x10\x00\x00\x01\x00\x06\x0A\x00\x00\x02\x00\x00\xB1\x00\x00\x03\x00\x00\xB2\x00\x00\x04\x00\x06%
ERROR: Found inconsistency in table tsdb
省略。。。
ERROR: Found lingering reference file hdfs://bdcluster/hbase/data/default/tsdb/3f67f9e2d0f154c71317947df8c286cb/t/d66557b8ad1048a0984217aab2f581e8.a099b47d2f958156a6700c6db9f55b38
ERROR: Found lingering reference file hdfs://bdcluster/hbase/data/default/tsdb/bd48d5231f4da0daccfe397747025df4/t/2e98baec0de0413eb5c8c8f32099f32e.b6a31dbe70640a08b003de588ba7bd97
ERROR: Found lingering reference file hdfs://bdcluster/hbase/data/default/tsdb/b0fb18494a411958a7add6e508967221/t/5b63ece8f7944146be55407200ca5f3d.a52c9d2cdb367b250cb251666d9007b4
ERROR: Found lingering reference file hdfs://bdcluster/hbase/data/default/tsdb/35f0bb902b1f1d20d90fe4cee919e6d8/t/577d2f0909094d578a6d13c127f5c4e3.2d5da6d0d97c96ba4a8d8c540db82b39
ERROR: Found lingering reference file hdfs://bdcluster/hbase/data/default/tsdb/404d67b43d0e0d1cfcb6754c0797cafd/t/64b7755b050e453dbdf684e1c407b7b0.382c8b2f50af99b56c68a065bf1ff630
ERROR: Found lingering reference file hdfs://bdcluster/hbase/data/default/tsdb/07dd208ad4a573c336cc8c64c2da85c7/t/50c4cf05e7d64f6e971b387b81d6367d.cf8dbd342a2870e0bd714e19bfe6aac2
ERROR: Found lingering reference file hdfs://bdcluster/hbase/data/default/tsdb/0f8b200c9ab044dc2c7ab6ef55188a7b/t/46a34c258d4944cdb3cbebd9a8f52f02.66c030248c6b0b07f670bb442ebeb1ad

hbase修复表

hbase hbck -fixReferenceFiles

8、hbase hbck检查报错

ERROR: Found lingering reference file hdfs://bdcluster/hbase/data/default/tsdb/

hbase修复表

hbase hbck -fixReferenceFiles

9、hbase hbck检查报错

Multiple regions have the same startkey：XXXX
There is an overlap in the region chain.

hbase修复表

hbase hbck -fixHdfsOverlaps

10、hbase hbck检查报错

found in META, but not in HDFS or deployed on any region server.
not deployed on any region server.
on HDFS, but not listed in hbase:meta or deployed on any region server.
There is a hole in the region chain between .You need to create a new .regioninfo and region dir in hdfs to plug the hole.

hbase修复表


hbase hbck -fixMeta -fixAssignments  -fixHdfsHoles

hbase hbck 修复过程详细解读

hbase hbck -fixAssignments
修复分布问题的时候输出

Trying to fix unassigned region...
2020-10-14 09:46:58,298 INFO  [hbasefsck-pool1-t48] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 09b306abe06352af702bcea64bb15f61, NAME => 'tsdb,\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5,1589273433052.09b306abe06352af702bcea64bb15f61.', STARTKEY => '\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5', ENDKEY => '\x00\x00(]\xE7\xC9\xF0\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04\x08'}
2020-10-14 09:46:58,298 INFO  [hbasefsck-pool1-t21] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 0358cf6973c12495d8a6100f4582f409, NAME => 'tsdb,\x00\x00`^\x12\xA2\xB0\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05*\x00\x00\x03\x00\x05+\x00\x00\x04\x00\x0Ag,1589367225052.0358cf6973c12495d8a6100f4582f409.', STARTKEY => '\x00\x00`^\x12\xA2\xB0\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05*\x00\x00\x03\x00\x05+\x00\x00\x04\x00\x0Ag', ENDKEY => '\x00\x00`^[O`\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05s\x00\x00\x03\x00\x05r\x00\x00\x04\x00\x0Ag'}
2020-10-14 09:46:58,298 INFO  [hbasefsck-pool1-t34] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 094c27940eeb70e774db3011b9b180fd, NAME => 'tsdb,\x00\x00\x06^\x85\x9B\x80\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x00K\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01/,1589377264511.094c27940eeb70e774db3011b9b180fd.', STARTKEY => '\x00\x00\x06^\x85\x9B\x80\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x00K\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01/', ENDKEY => '\x00\x00\x06^\x87\xB1\xE0\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x004\x00\x00\x03\x00\x00&\x00\x00\x04\x00\x01''}
2020-10-14 09:46:58,298 INFO  [hbasefsck-pool1-t20] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 083a0ef5b0ba63cf4fbc496d027cfaa2, NAME => 'tsdb,\x00\x00\x06]_\x9D\x00\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x01I\x00\x00\x03\x00\x00\x10\x00\x00\x04\x00\x01-,1589377409936.083a0ef5b0ba63cf4fbc496d027cfaa2.', STARTKEY => '\x00\x00\x06]_\x9D\x00\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x01I\x00\x00\x03\x00\x00\x10\x00\x00\x04\x00\x01-', ENDKEY => '\x00\x00\x06]a\xA5P\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x02\xA3\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01$'}
2020-10-14 09:47:43,678 INFO  [hbasefsck-pool1-t14] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 037ba0dcb69e2453ae17b04682a0607a, NAME => 'tsdb,\x00\x005]\xB8\x0D\xA0\x00\x00\x01\x00\x04W\x00\x00\x02\x00\x0C$,1589284372576.037ba0dcb69e2453ae17b04682a0607a.', STARTKEY => '\x00\x005]\xB8\x0D\xA0\x00\x00\x01\x00\x04W\x00\x00\x02\x00\x0C$', ENDKEY => '\x00\x005]\xB9\xCF\xA0\x00\x00\x01\x00\x04W\x00\x00\x02\x00\x0C('}
2020-10-14 09:47:43,681 INFO  [hbasefsck-pool1-t28] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 09c15fdb7274bc56650d40d2f8a82adc, NAME => 'tsdb,\x00\x00&\x5C\x0A\x1Ap\x00\x00\x01\x00\x05\x9D\x00\x00\x02\x00\x03\xF9\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x05\xCB,1589376226633.09c15fdb7274bc56650d40d2f8a82adc.', STARTKEY => '\x00\x00&\x5C\x0A\x1Ap\x00\x00\x01\x00\x05\x9D\x00\x00\x02\x00\x03\xF9\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x05\xCB', ENDKEY => '\x00\x00&\x5C\x0BA\xC0\x00\x00\x01\x00\x05\x9D\x00\x00\x02\x00\x00\x9F\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x05\xCE'}
2020-10-14 09:47:50,620 INFO  [hbasefsck-pool1-t21] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 0358cf6973c12495d8a6100f4582f409, NAME => 'tsdb,\x00\x00`^\x12\xA2\xB0\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05*\x00\x00\x03\x00\x05+\x00\x00\x04\x00\x0Ag,1589367225052.0358cf6973c12495d8a6100f4582f409.', STARTKEY => '\x00\x00`^\x12\xA2\xB0\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05*\x00\x00\x03\x00\x05+\x00\x00\x04\x00\x0Ag', ENDKEY => '\x00\x00`^[O`\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05s\x00\x00\x03\x00\x05r\x00\x00\x04\x00\x0Ag'}
2020-10-14 09:47:50,620 INFO  [hbasefsck-pool1-t48] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 09b306abe06352af702bcea64bb15f61, NAME => 'tsdb,\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5,1589273433052.09b306abe06352af702bcea64bb15f61.', STARTKEY => '\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5', ENDKEY => '\x00\x00(]\xE7\xC9\xF0\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04\x08'}
2020-10-14 09:47:50,620 INFO  [hbasefsck-pool1-t34] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 094c27940eeb70e774db3011b9b180fd, NAME => 'tsdb,\x00\x00\x06^\x85\x9B\x80\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x00K\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01/,1589377264511.094c27940eeb70e774db3011b9b180fd.', STARTKEY => '\x00\x00\x06^\x85\x9B\x80\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x00K\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01/', ENDKEY => '\x00\x00\x06^\x87\xB1\xE0\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x004\x00\x00\x03\x00\x00&\x00\x00\x04\x00\x01''}
2020-10-14 09:47:50,621 INFO  [hbasefsck-pool1-t19] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 0927b941803c51576584ff1b6eaa8891, NAME => 'tsdb,\x00\x00d\x5CB"0\x00\x00\x01\x00\x0A\x98\x00\x00\x02\x00\x05\x0D\x00\x00\x03\x00\x03\xD7\x00\x00\x04\x00\x0A\xA5,1589289484255.0927b941803c51576584ff1b6eaa8891.', STARTKEY => '\x00\x00d\x5CB"0\x00\x00\x01\x00\x0A\x98\x00\x00\x02\x00\x05\x0D\x00\x00\x03\x00\x03\xD7\x00\x00\x04\x00\x0A\xA5', ENDKEY => '\x00\x00d\x5CB0@\x00\x00\x01\x00\x0A\x98\x00\x00\x02\x00\x04\xF5\x00\x00\x03\x00\x04\xF6\x00\x00\x04\x00\x0A\xAB'}
2020-10-14 09:47:50,621 INFO  [hbasefsck-pool1-t43] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 040f247ad3c3f4191c11d261b8082879, NAME => 'tsdb,\x00\x00_^\xB9\xE7\x00\x00\x00\x01\x00\x0A@\x00\x00\x02\x00\x04\x02\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x0AJ,1602581836946.040f247ad3c3f4191c11d261b8082879.', STARTKEY => '\x00\x00_^\xB9\xE7\x00\x00\x00\x01\x00\x0A@\x00\x00\x02\x00\x04\x02\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x0AJ', ENDKEY => '\x00\x00_^\xFE\xC9`\x00\x00\x01\x00\x0A@\x00\x00\x02\x00\x04\x04\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x0AJ'}
2020-10-14 09:47:51,620 WARN  [hbasefsck-pool1-t21] util.HBaseFsck: Unable to complete check or repair the region 'tsdb,\x00\x00`^\x12\xA2\xB0\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05*\x00\x00\x03\x00\x05+\x00\x00\x04\x00\x0Ag,1589367225052.0358cf6973c12495d8a6100f4582f409.'.
java.io.IOException: Region {ENCODED => 0358cf6973c12495d8a6100f4582f409, NAME => 'tsdb,\x00\x00`^\x12\xA2\xB0\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05*\x00\x00\x03\x00\x05+\x00\x00\x04\x00\x0Ag,1589367225052.0358cf6973c12495d8a6100f4582f409.', STARTKEY => '\x00\x00`^\x12\xA2\xB0\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05*\x00\x00\x03\x00\x05+\x00\x00\x04\x00\x0Ag', ENDKEY => '\x00\x00`^[O`\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05s\x00\x00\x03\x00\x05r\x00\x00\x04\x00\x0Ag'} failed to move out of transition within timeout 120000ms
	at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:149)
	at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:2142)
	at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:2343)
	at org.apache.hadoop.hbase.util.HBaseFsck.access$1100(HBaseFsck.java:195)
	at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1915)
	at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1903)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-10-14 09:47:51,626 WARN  [hbasefsck-pool1-t48] util.HBaseFsck: Unable to complete check or repair the region 'tsdb,\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5,1589273433052.09b306abe06352af702bcea64bb15f61.'.
java.io.IOException: Region {ENCODED => 09b306abe06352af702bcea64bb15f61, NAME => 'tsdb,\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5,1589273433052.09b306abe06352af702bcea64bb15f61.', STARTKEY => '\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5', ENDKEY => '\x00\x00(]\xE7\xC9\xF0\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04\x08'} failed to move out of transition within timeout 120000ms
	at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:149)
	at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:2142)
	at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:2343)
	at org.apache.hadoop.hbase.util.HBaseFsck.access$1100(HBaseFsck.java:195)
	at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1915)
	at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1903)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-10-14 09:47:51,627 WARN  [hbasefsck-pool1-t48] util.HBaseFsck: Skip region 'tsdb,\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5,1589273433052.09b306abe06352af702bcea64bb15f61.'
ERROR: Region { meta => tsdb,\x00\x00\x0C\x5C\xE0\x01\xD0\x00\x00\x01\x00\x02\x05\x00\x00\x02\x00\x01\x82\x00\x00\x03\x00\x00\x10\x00\x00\x04\x00\x02\x1B,1589397686856.0c308622a2586fa038579ae2572d8f70., hdfs => hdfs://bdcluster/hbase/data/default/tsdb/0c308622a2586fa038579ae2572d8f70, deployed => , replicaId => 0 } not deployed on any region server.
Trying to fix unassigned region...
2020-10-14 09:47:51,626 WARN  [hbasefsck-pool1-t34] util.HBaseFsck: Unable to complete check or repair the region 'tsdb,\x00\x00\x06^\x85\x9B\x80\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x00K\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01/,1589377264511.094c27940eeb70e774db3011b9b180fd.'.
java.io.IOException: Region {ENCODED => 094c27940eeb70e774db3011b9b180fd, NAME => 'tsdb,\x00\x00\x06^\x85\x9B\x80\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x00K\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01/,1589377264511.094c27940eeb70e774db3011b9b180fd.', STARTKEY => '\x00\x00\x06^\x85\x9B\x80\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x00K\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01/', ENDKEY => '\x00\x00\x06^\x87\xB1\xE0\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x004\x00\x00\x03\x00\x00&\x00\x00\x04\x00\x01''} failed to move out of transition within timeout 120000ms
	at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:149)
	at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:2142)
	at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:2343)
	at org.apache.hadoop.hbase.util.HBaseFsck.access$1100(HBaseFsck.java:195)
	at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1915)
	at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1903)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-10-14 09:47:51,628 WARN  [hbasefsck-pool1-t34] util.HBaseFsck: Skip region 'tsdb,\x00\x00\x06^\x85\x9B\x80\x00\x00\x01\x00\x01"\x00\x00\x02\x00\x00K\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01/,1589377264511.094c27940eeb70e774db3011b9b180fd.'
2020-10-14 09:47:51,626 WARN  [hbasefsck-pool1-t21] util.HBaseFsck: Skip region 'tsdb,\x00\x00`^\x12\xA2\xB0\x00\x00\x01\x00\x05\x04\x00\x00\x02\x00\x05*\x00\x00\x03\x00\x05+\x00\x00\x04\x00\x0Ag,1589367225052.0358cf6973c12495d8a6100f4582f409.'
ERROR: Region { meta => tsdb,\x00\x00a^\xB9\xBC\xD0\x00\x00\x01\x00\x0Ah\x00\x00\x02\x00\x00p\x00\x00\x03\x00\x00q\x00\x00\x04\x00\x0Al,1602577336806.0c72e1d1db96802e4823f0ede448429f., hdfs => hdfs://bdcluster/hbase/data/default/tsdb/0c72e1d1db96802e4823f0ede448429f, deployed => , replicaId => 0 } not deployed on any region server.
Trying to fix unassigned region...
ERROR: Region { meta => tsdb,\x00\x00\x0B^\xB1\xB7\xB0\x00\x00\x01\x00\x01\xE4\x00\x00\x02\x00\x00H\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x01\xEC,1589355732620.0ca3a2bf4cf091e97dbe26624c9757e6., hdfs => hdfs://bdcluster/hbase/data/default/tsdb/0ca3a2bf4cf091e97dbe26624c9757e6, deployed => , replicaId => 0 } not deployed on any region server.
Trying to fix unassigned region...
2020-10-14 09:47:58,791 INFO  [hbasefsck-pool1-t28] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 09c15fdb7274bc56650d40d2f8a82adc, NAME => 'tsdb,\x00\x00&\x5C\x0A\x1Ap\x00\x00\x01\x00\x05\x9D\x00\x00\x02\x00\x03\xF9\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x05\xCB,1589376226633.09c15fdb7274bc56650d40d2f8a82adc.', STARTKEY => '\x00\x00&\x5C\x0A\x1Ap\x00\x00\x01\x00\x05\x9D\x00\x00\x02\x00\x03\xF9\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x05\xCB', ENDKEY => '\x00\x00&\x5C\x0BA\xC0\x00\x00\x01\x00\x05\x9D\x00\x00\x02\x00\x00\x9F\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x05\xCE'}
2020-10-14 09:47:58,791 INFO  [hbasefsck-pool1-t14] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 037ba0dcb69e2453ae17b04682a0607a, NAME => 'tsdb,\x00\x005]\xB8\x0D\xA0\x00\x00\x01\x00\x04W\x00\x00\x02\x00\x0C$,1589284372576.037ba0dcb69e2453ae17b04682a0607a.', STARTKEY => '\x00\x005]\xB8\x0D\xA0\x00\x00\x01\x00\x04W\x00\x00\x02\x00\x0C$', ENDKEY => '\x00\x005]\xB9\xCF\xA0\x00\x00\x01\x00\x04W\x00\x00\x02\x00\x0C('}
省略。。。
2020-10-14 10:11:22,095 WARN  [hbasefsck-pool1-t16] util.HBaseFsck: Unable to complete check or repair the region 'tsdb,\x00\x00\x05^A7\xB0\x00\x00\x01\x00\x00\xFA\x00\x00\x02\x00\x00?\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x00\xFC,1589394403198.154794a27addf79de866da8693b03c28.'.
java.io.IOException: Region {ENCODED => 154794a27addf79de866da8693b03c28, NAME => 'tsdb,\x00\x00\x05^A7\xB0\x00\x00\x01\x00\x00\xFA\x00\x00\x02\x00\x00?\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x00\xFC,1589394403198.154794a27addf79de866da8693b03c28.', STARTKEY => '\x00\x00\x05^A7\xB0\x00\x00\x01\x00\x00\xFA\x00\x00\x02\x00\x00?\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x00\xFC', ENDKEY => '\x00\x00\x05^C\x15\xD0\x00\x00\x01\x00\x00\xFA\x00\x00\x02\x00\x03\x0E'} failed to move out of transition within timeout 120000ms
	at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:149)
	at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:2142)
	at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:2343)
	at org.apache.hadoop.hbase.util.HBaseFsck.access$1100(HBaseFsck.java:195)
	at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1915)
	at org.apache.hadoop.hbase.util.HBaseFsck$CheckRegionConsistencyWorkItem.call(HBaseFsck.java:1903)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-10-14 10:11:22,095 WARN  [hbasefsck-pool1-t16] util.HBaseFsck: Skip region 'tsdb,\x00\x00\x05^A7\xB0\x00\x00\x01\x00\x00\xFA\x00\x00\x02\x00\x00?\x00\x00\x03\x00\x00<\x00\x00\x04\x00\x00\xFC,1589394403198.154794a27addf79de866da8693b03c28.'
ERROR: Region { meta => tsdb,\x00\x00_^\xFE\xC9`\x00\x00\x01\x00\x0A@\x00\x00\x02\x00\x04\x04\x00\x00\x03\x00\x03\xF8\x00\x00\x04\x00\x0AJ,1602581836946.2bb46a923325fb29a936e83465887eb3., hdfs => hdfs://bdcluster/hbase/data/default/tsdb/2bb46a923325fb29a936e83465887eb3, deployed => , replicaId => 0 } not deployed on any region server.
Trying to fix unassigned region...

说明：执行fixAssigment会循环修复未被deploy的表。
注意日志中的输出

INFO  [hbasefsck-pool1-t48] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 09b306abe06352af702bcea64bb15f61, NAME => 'tsdb,\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5,1589273433052.09b306abe06352af702bcea64bb15f61.', STARTKEY => '\x00\x00(]\x1BG@\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04/\x00\x00\x03\x00\x03\x91\x00\x00\x04\x00\x05\xE5', ENDKEY => '\x00\x00(]\xE7\xC9\xF0\x00\x00\x01\x00\x05\xDF\x00\x00\x02\x00\x04\x08'}

这是由于hbase的Region状态迁移，观察hbase master（60010）界面会发现。

Regions in Transition
Region	                        State	                                  RIT time (ms)
e12f459fc7da3e5937551a2f9484cbbd	tsdb,\x00\x00\x02^q\xD5\x00\x00\x00\x01\x00\x00\x97\x00\x00\x02\x00\x03\x09\x00\x00\x03\x00\x00&\x00\x00\x04\x00\x06\xCD,1589279163414.e12f459fc7da3e5937551a2f9484cbbd. state=PENDING_OPEN, ts=Wed Oct 14 09:06:28 CST 2020 (4117s ago), server=c7003,16020,1602637554384	4117515
省略。。。
Total number of Regions in Transition for more than 60000 milliseconds	141	
Total number of Regions in Transition	192	
(92 more regions in transition not shown)

随着时间推移这个Total number of Regions in Transition的数字会自动减少。
详细的region 状态迁移讲解请参考这篇博文

https://blog.csdn.net/lw_ghy/article/details/60780065

第2章 hbase修复详解

以下为摘抄自别人博客上的内容
新版本的 hbck 可以修复各种错误，修复选项是：

（1）-fix，向下兼容用，被-fixAssignments替代     
（2）-fixAssignments，用于修复region assignments错误     
（3）-fixMeta，用于修复meta表的问题，前提是HDFS上面的region info信息有并且正确。     
（4）-fixHdfsHoles，修复region holes（空洞，某个区间没有region）问题     
（5）-fixHdfsOrphans，修复Orphan region（hdfs上面没有.regioninfo的region）     
（6）-fixHdfsOverlaps，修复region overlaps（区间重叠）问题     
（7）-fixVersionFile，修复缺失hbase.version文件的问题     
（8）-maxMerge <n> （n默认是5），当region有重叠是，需要合并region，一次合并的region数最大不超过这个值。     
（9）-sidelineBigOverlaps ，当修复region overlaps问题时，允许跟其他region重叠次数最多的一些region不参与（修复后，可以把没有参与的数据通过bulk load加载到相应的region）     
（10）-maxOverlapsToSideline <n> （n默认是2），当修复region overlaps问题时，一组里最多允许多少个region不参与     
由于选项较多，所以有两个简写的选项     
（11） -repair，相当于-fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps     
（12）-repairHoles，相当于-fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans

新版本的 hbck

（1）缺失hbase.version文件     
 加上选项 -fixVersionFile 解决     
（2）如果一个region即不在META表中，又不在hdfs上面，但是在regionserver的online region集合中     
 加上选项 -fixAssignments 解决     
（3）如果一个region在META表中，并且在regionserver的online region集合中，但是在hdfs上面没有     
 加上选项 -fixAssignments -fixMeta 解决，（ -fixAssignments告诉regionserver close region），（ -fixMeta删除META表中region的记录）     
（4）如果一个region在META表中没有记录，没有被regionserver服务，但是在hdfs上面有     
加上选项 -fixMeta -fixAssignments 解决，（ -fixAssignments 用于assign region），（ -fixMeta用于在META表中添加region的记录）     
（5）如果一个region在META表中没有记录，在hdfs上面有，被regionserver服务了     
加上选项 -fixMeta 解决，在META表中添加这个region的记录，先undeploy region，后assign     
（6）如果一个region在META表中有记录，但是在hdfs上面没有，并且没有被regionserver服务     
加上选项 -fixMeta 解决，删除META表中的记录     
（7）如果一个region在META表中有记录，在hdfs上面也有，table不是disabled的，但是这个region没有被服务     
加上选项 -fixAssignments 解决，assign这个region     
（8）如果一个region在META表中有记录，在hdfs上面也有，table是disabled的，但是这个region被某个regionserver服务了     
加上选项 -fixAssignments 解决，undeploy这个region     
（9）如果一个region在META表中有记录，在hdfs上面也有，table不是disabled的，但是这个region被多个regionserver服务了     
加上选项 -fixAssignments 解决，通知所有regionserver close region，然后assign region     
（10）如果一个region在META表中，在hdfs上面也有，也应该被服务，但是META表中记录的regionserver和实际所在的regionserver不相符     
加上选项 -fixAssignments 解决     

（11）region holes     
需要加上 -fixHdfsHoles ，创建一个新的空region，填补空洞，但是不assign 这个 region，也不在META表中添加这个region的相关信息     
（12）region在hdfs上面没有.regioninfo文件     
-fixHdfsOrphans 解决     
（13）region overlaps     
需要加上 -fixHdfsOverlaps

说明：

（1）修复region holes时，-fixHdfsHoles 选项只是创建了一个新的空region，填补上了这个区间，还需要加上-fixAssignments -fixMeta 来解决问题，（ -fixAssignments 用于assign region），（ -fixMeta用于在META表中添加region的记录），所以有了组合拳 -repairHoles 修复region holes，相当于-fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans     
（2） -fixAssignments，用于修复region没有assign、不应该assign、assign了多次的问题     
（3）-fixMeta，如果hdfs上面没有，那么从META表中删除相应的记录，如果hdfs上面有，在META表中添加上相应的记录信息     
（4）-repair 打开所有的修复选项，相当于-fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps     

新版本的hbck从（1）hdfs目录（2）META（3）RegionServer这三处获得region的Table和Region的相关信息，根据这些信息判断并repair

hbase报错web ui报错

在这里插入图片描述

The Load Balancer is not enabled which will eventually cause performance degradation in HBase as Regions will not be distributed across all RegionServers. The balancer is only expected to be disabled during rolling upgrade scenarios.

在HBase shell中执行，balance_switch true 启动平衡器后，再次查看HBase Web UI，警告消失。

Region splits are disabled. This may be the result of HBCK aborting while running in repair mode. Manually enable splits from the HBase shell, or re-run HBCK in repair mode.

在HBase shell中执行splitormerge_enabled ‘SPLIT’

hbase(main):030:0> splitormerge_enabled ‘MERGE’

Region merges are disabled. This may be the result of HBCK aborting while running in repair mode. Manually enable merges from the HBase shell, or re-run HBCK in repair mode.

在HBase shell中执行splitormerge_enabled ‘MERGE’

时空无限

关注

1
点赞
踩
14

收藏

觉得还不错? 一键收藏
打赏
0
评论
hbase表修复

场景opentsdb启动报错grep -i error /tmp/opentsdb.log2020-10-13 12:19:41,098 INFO [main-SendThread(localhost:2181)] ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)2020-10-13
复制链接

扫一扫