记一次HBASE META表故障

最新推荐文章于 2024-04-23 10:18:51 发布

xtajmh00

最新推荐文章于 2024-04-23 10:18:51 发布

阅读量3.8k

点赞数

本文链接：https://blog.csdn.net/xtajmh00/article/details/52813967

版权

本文记录了一次HBASE集群中遇到的META表故障问题，详细描述了17号节点regionserver的日志错误，包括ZooKeeper连接丢失异常。在尝试关闭并重新平衡regionserver、执行hbck命令等操作后，发现仍有region未正确部署。最终通过手动assign解决部分问题，但仍有4个异常未能解决，计划通过重启HBASE集群来解决剩余问题。

摘要由CSDN通过智能技术生成

HBASE菜鸟一枚，欢迎吐槽

hbase版本：0.90.6

共32个节点

17号节点regionserver日志：

2016-10-12 18:02:07,051 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: regionserver:60020-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009 Unable to get data of znode /hbase/unassigned/e50b05f8446a86044f32d64fd92df5ee
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/unassigned/e50b05f8446a86044f32d64fd92df5ee
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:614)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:707)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpening(ZKAssign.java:589)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpening(ZKAssign.java:582)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.transitionZookeeperOfflineToOpening(OpenRegionHandler.java:354)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:96)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:162)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
2016-10-12 18:02:07,051 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: regionserver:60020-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009-0x453857482f10009 Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/unassigned/e50b05f8446a86044f32d64fd92df5ee
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:614)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:707)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpening(ZKAssign.java:589)
        at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpening(ZKAssign.java:582)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.transitionZookeeperOfflineToOpening(OpenRegionHandler.java:354)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:96)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:162)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

2016-10-12 18:02:07,051 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegi