hadoop集群运行异常故障描述

今天上午在正常操作hadoop的时候，突然发现所有的节点都连不上，然后在hadoop的简单管理页面上所有节点都是Dead Nodes。

检查各节点日志发现：namenode和jobtracker都连不上了，各节点报错信息如下：

java.net.ConnectException: Call to master/XXX.XXX.XXX.XXX:XXXX failed on connection exception: java.net.ConnectException: Connection refused
        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1099)
        at org.apache.hadoop.ipc.Client.call(Client.java:1075)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
        at $Proxy5.sendHeartbeat(Unknown Source)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:904)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1458)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
        at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
        at org.apache.hadoop.ipc.Client.call(Client.java:1050)
        ... 5 more

 

然后大概过了20几分钟后，节点全部自动重连，在管理页面上看见所有的节点都是Live Nodes。

暂不知道是什么问题导致。