一大早起来关注我的小集群,发现HBase不好使了,弄得毫无头绪——只好重新查看守护进程jps 后 返现传说中的HMaster居然。。。不见了!
————查看日志!果然:
2012-12-17 22:51:13,317 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3ba92b93c80000 has expired, closing socket connection
2012-12-17 22:51:13,318 FATAL org.apache.hadoop.hbase.master.HMaster: master:60000-0x3ba92b93c80000 master:60000-0x3ba92b93c80000 received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:344)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:262)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
2012-12-17 22:51:13,319 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
哈哈,这下就清楚了,google一下“regionServer宕机” Zookeeper service session expired
有人做出如下解释,挺好的:
网络断开,心跳发送失败,尝试连接其他的zookeeper服务器。(zookeeper会尝试连接其他所有的服务器), 网络恢复了,连接成功,但 session已经过期了,所以 zookeeper 客户端关闭了;当然HMaster也会受到Zookeeper的过期失效信息,产生中断;
相关重要原因总结如下:本人小集群全部虚拟机,网络要考虑;一些设置参数,没有仔细弄清楚,也必须重新审视;
解决方案以及源代码的审视:http://blog.sina.com.cn/s/blog_6b10e1740100rzi0.html http://jiajun.iteye.com/blog/1013215 以及 http://blog.csdn.net/ucool2007/article/details/6604612 其中文章http://blog.sina.com.cn/s/blog_6b10e1740100rzi0.html还对java中的GC机制进行了考虑,给出了配置文件