注: 转自 HBase region is not online 问题修复
一年多没有搞HBase了,回想前年和营神一起战斗的日子,~~~。今天线上遇到下面一个问题:
hbase(main):002:0> get 'mynamespace:user_basic_info','BAC3510A922CF026500874EA3975E123'
COLUMN CELL
ERROR: org.apache.hadoop.hbase.NotServingRegionException: Region mynamespace:user_basic_info,BA5968E36ADB91CE1EA37D44267F5865,1489326561674.0250284baa6119d676821e86cfaa29f4. is not online on *******,60020,1491385979553
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2922)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1053)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2006)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
其他region 都是正常的,重启regionserver 后依然报同样的错误。
首先检查这张表是否存储一致性问题
hbase hbck -details table
- 发现的确出现了2个不一致的地方
2 inconsistencies detected.
既然不一致,咱就尝试修复一下:
hbase hbck -repair table
这个功能要管理权限,使用慎重!修复完了以后结果如下
Summary:
Table hbase:meta is okay.
Number of regions: 1
Deployed on: ctum2f0602005.idc.wanda-group.net,60020,1482504754412
Table idctag:user_basic_info is okay.
Number of regions: 124
0 inconsistencies detected.
Status: OK