standby导致spark初始化失败问题记录

问题：

11:15:50,290 ERROR ~ Error initializing SparkContext.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1565)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1181)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3463)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:764)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:764)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

 

解决办法1：切换状态

[hadoop@hadoop1 conf]$ hdfs haadmin -getServiceState nn1
standby
[hadoop@hadoop1 conf]$ hdfs haadmin -getServiceState nn2
active
[hadoop@hadoop1 conf]$ hdfs haadmin -transitionToActive nn1
Automatic failover is enabled for NameNode at hadoop1/192.168.0.165:53310
Refusing to manually manage HA state, since it may cause
a split-brain scenario or other incorrect state.
If you are very sure you know what you are doing, please 
specify the forcemanual flag.
[hadoop@hadoop1 conf]$ hdfs haadmin -getServiceState nn2
standby
[hadoop@hadoop1 conf]$ hdfs haadmin -getServiceState nn1
active

 

或者：

hdfs  haadmin -failover --forcefence --forceactive  nn2  nn1
注意：此处“nn2 nn1”的顺序表示active状态由nn2转换到nn1上（虽然nn2在转化前也是standby状态）。
但若在hdfs-site.xml中，配置：
<property>
     <name>dfs.ha.automatic-failover.enabled.hdpcdh4</name>
     <value>true</value>
</property>

就不能手动切换了，直接到nn2上kill掉NameNode就行了

 

办法二（没试过）：

参考：https://blog.csdn.net/qq_32253371/article/details/77008544

yarn-site.xml配置

<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>