HBase查看日志报错 msg=拒绝连接 row ‘‘ on table ‘hbase:meta‘ at region=hbase:meta,,1.1588230740, hostname=xxx

今天在hbase shell中执行一些语句时发现特别卡顿，然后查看了一下日志，发现报错：

2021-11-30 14:15:58,777 INFO  [htable-pool17-t1] client.RpcRetryingCaller: Call exception, tries=11, retries=350, started=48510 ms ago, cancelled=false, msg=拒绝连接 row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=xxx,16020,1638250799478, seqNum=0
2021-11-30 14:15:58,778 WARN  [xxx,16000,1638250798693_ChoreService_1] master.CatalogJanitor: Failed scan of catalog table
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=351, exceptions:
Tue Nov 30 14:15:58 CST 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68541: 拒绝连接 row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=xxx,16020,1638250799478, seqNum=0

        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:164)
        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:159)
        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:796)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
        at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:183)
        at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:135)
        at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:237)
        at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:117)
        at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:189)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68541: 拒绝连接 row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=linux123,16020,1638250799478, seqNum=0
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:171)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
        ... 3 more

原因及解决方案

原来是hbase集群中的一台机器的hbase服务下线了，重新启动那台机器的hbase服务，问题解决。