1. HDFS相关问题
症状1:HDFS HA场景下,启动时出现两个NN都为standby
Namenode日志中提示:
WARN org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a roll of the active NN
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category JOURNAL is not supported in state standby
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1691)
......
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
Failover Controller日志中提示:
WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
java.lang.IllegalArgumentException: Unable to determine service address for namenode 'namenode123'
at org.apache.hadoop.hdfs.tools.NNHAServiceTarget.<init>(NNHAServiceTarget.java:76)
at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.dataToTarget(DFSZKFailoverController.java:69)
......
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
版本:2.5.0+cdh5.2.0
原因分析:
此类问题一般是由ZKFC引起。此时ZK将会有一个ZNode:/hadoop-ha/${dfs.nameservices}/ActiveBreadCrumb,查看其内容应该会发现namenode123关键字。这表明ZKFC在寻找nameservice ID为namenode123的NN,但是本次启动时NN的nameservice ID却不是namenode123(本次nameservice ID可在hdfs-site.xml中看到)。暂不清楚ZKFC为何会检测一个旧的nameservice ID。
解决办法:
在ZK中初始化HA状态,可通过CM界面操作,或者执行sudo -u hdfs hdfs zkfc -formatZK,然后重启HDFS即可。
症状1:NN异常的接收到SIGNAL 15,从而进程退出
Namenode日志中除了如下提示之外,别无异常:
ERROR