现象为:python happybase连接hbase卡死,查看thrift里的日志发现:
2018-06-26 15:19:43,192 INFO [thrift-worker-11] client.RpcRetryingCaller: Call exception, tries=21, retries=35, started=249681 ms ago, cancelled=false, msg=row '424998_bugzilla.intra.nsfocus.com_10.66.104.1_192.168.11.251_80_3600' on table 'web_profiling_user' at region=web_profiling_user,,1529991310552.88063c7728b6e892d914254fba33906e., hostname=bsa151,16020,1529996486170, seqNum=4
2018-06-26 15:20:02,573 INFO [thrift-worker-12] client.RpcRetryingCaller: Call exception, tries=21, retries=35, started=249213 ms ago, cancelled=false, msg=row '424998_bugzilla.intra.nsfocus.com_1_192.168.11.251_80_3600' on table 'web_profiling_user' at region=web_profiling_user,,1529991310552.88063c7728b6e892d914254fba33906e., hostname=bsa151,16020,1529996486170, seqNum=4
2018-06-26 15:20:02,716 INFO [thrift-worker-0] client.RpcRetryingCaller: Call exception, tries=21, retries=35, started=249354 ms ago, cancelled=false, msg=row '424998_bugzilla.intra.nsfocus.com_1_192.168.11.251_80_3600' on table 'web_profiling_user' at region=web_profiling_user,,1529991310552.88063c7728b6e892d914254fba33906e., hostname=bsa151,16020,1529996486170, seqNum=4
同时伴随报错:
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: Region web_profiling_user,,1529991310552.88063c7728b6e892d914254fba33906e. is not online on bsa151,16020,1529994706603
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2942)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1072)
此时,可以使用:
./hbase hbck -details web_profiling_user
命令查看表的情况,如果发现存在:
inconsistent后面不是0的信息,则表示表被破坏了。
可以使用:
./hbase hbck -repair web_profiling_user
修复表。