Hadoop & HBase 故障排除

Hadoop & HBase 故障排除

接到报警反馈HBase故障。

1.上线检查HBase日志信息，有如下报错信息，无法链接RegionServer

2015-06-23 08:50:39,219 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInM
eta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=Colander-hb03, port=60020}, attempt=81 of 100 fa
iled; retrying after sleep of 32044 because: Call to Colander-hb03/10.161.181.208:60020 failed on local exception: java.io
.IOException: Connection reset by peer

2.决定重启HBase，执行stop-hbase.sh很长时间无法关闭，使用kill杀掉相关进程，再start-hbase.sh开启HBase。

3.开启后无法正常工作，日志未出现任何错误，HMaster显示available，但是RegionServer显示master not available

HMaster日志

2015-06-23 09:41:43,644 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: A master is now available
2015-06-23 09:41:43,651 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=Colander-hb01,60000,1435023696774

RegionServer日志

2015-06-23 09:55:08,156 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master isn’t available yet, retrying
2015-06-23 09:55:08,357 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at Colander-hb01,60000,1435024143478

4.HMaster 和 RegionServer 都运行正常，问题调查无法进展下去。换个思路，看是否HDFS故障。

5.检查Hadoop日志，发现确实有问题

2015-06-23 09:40:36,292 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.168.234.177:50010, storageID=DS-1368639422-10.168.234.177-50010-1417075244953, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:295)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:339)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:403)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:581)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:406)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
at java.lang.Thread.run(Thread.java:745)

6.重启Hadoop DFS

./bin/stop-dfs.sh; ./bin/start-dfs.sh

7.HBase依然运行异常，重启后恢复正常。