意外停电了几分钟,重启hadoop集群后,hdfs系统不正常。发现所有的DataNode都无法连接NameNode。
DataNode日志中出现:
2014-02-23 12:15:25,378 INFO org.apache.hadoop.ipc.RPC: Server at NameNode01/192.168.40.236:9000 not available yet, Zzzzz...
2014-02-23 12:15:27,378 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:28,379 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:29,379 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:30,380 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:31,380 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:32,381 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:33,381 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:34,382 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:35,382 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2014-02-23 12:15:36,382 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: NameNode01/192.168.40.236:9000. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
NameNode日志出现:
2014-02-23 13:09:57,681 WARN org.apache.hadoop.hdfs.server.namenode.FSEditLog: Cannot roll edit log, edits.new files already exists in all healthy directories:
/opt/hadoop-1.2.1/hadoop/hdfs/name/current/edits.new
无法正常查看。
在NameNode上尝试性的执行Hadoop fs -ls / 后,出现:
WARN ipc.Client: Address change detected. Old: NameNode01/192.168.40.236:9000 New: NameNode01/127.0.1.1:9000
解决:
删除NameNode节点上/etc/hosts中127.0.0.1 NameNode的配置后,重启恢复正常。