[HADOOP]我所遇到的Hadoop报错(更新中)

我得集群上安装了三套hadoop，所以冲突的时候也属于意料之中了。

hadoop的fs命令官网地址

http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.19.1/docs/cn/hdfs_shell.html

格式化namenode报错

删除namenode格式化所需要目录

 rm -rf /tmp/hadoop/dfs/journalnode/mycluster/*

这样就可以显示succeefully

然后进入bin目录

bin/hdfs namenode -format

会出现successfully就是ok了

hdfs的命令报错

HA模式如果重启zookeeper经常报如下错误：

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over vm-CentOS75-0-149/192.168.0.149:9000 after 3 failover attempts. Trying to failover after sleeping for 2285ms.
22/07/27 10:54:12 INFO retry.RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2021)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1449)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3183)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:979)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over vm-CentOS75-0-150/192.168.0.150:9000 after 4 failover attempts. Trying to failover after sleeping for 4276ms.

2022-06-21 15:50:02,529 INFO retry.RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2021)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1449)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3183)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1173)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:979)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over vm-centos75-153/192.168.0.153:9000 after 1 failover attempts. Trying to failover after sleeping for 706ms.

问题就是namenode节点都是standby状态，需要下面的命令切换到active状态

原因：由于配置了ha模式，执行命令的机器不是active节点
下面是换成第一个节点，执行以下命令，

[root@vm-centos75-154 bdp]# ./hdfs/bin/hdfs haadmin -transitionToStandby --forcemanual nn1
[root@vm-centos75-154 bdp]# ./hdfs/bin/hdfs haadmin -transitionToActive --forcemanual nn1

输入Y。
然后切换到nn1的节点

阿这，不报错了。

查看HADOOP的HA模式RM的active节点

/opt/bdp/yarn/bin/yarn rmadmin -getAllServiceState   

切换

/opt/bdp/yarn/bin/yarn rmadmin -transitionToStandby rm1

所有节点突然变成standby

查看日志，发现是hadoop的HA模式下需要zkfc来跟namenode节点联络。查看日志发现zkfc未启动。
因为zkfc目录消失，需要重新格式化：

/opt/bdp/hdfs/bin/hdfs zkfc -formatZK

然后启动zkfc
namenode节点从standby变为active

……
2022-07-29 11:17:05,161 INFO org.apache.zookeeper.ZooKeeper: Session: 0x3000f6e84850001 closed
2022-07-29 11:17:05,165 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x3000f6e84850001
2022-07-29 11:17:05,168 INFO org.apache.hadoop.hdfs.tools.DFSZKFailoverController: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at vm-CentOS75-0-149/192.168.0.149
************************************************************/
[root@vm-CentOS75-0-149 hadoop]# /opt/bdp/hdfs/
bin/         include/     libexec/     logs/        README.txt   share/
etc/         lib/         LICENSE.txt  NOTICE.txt   sbin/
[root@vm-CentOS75-0-149 hadoop]# /opt/bdp/hdfs/bin/hdfs
hdfs      hdfs.cmd
[root@vm-CentOS75-0-149 hadoop]# /opt/bdp/hdfs/bin/hdfs zkfc -formatZK
zkfc is running as process 27341.  Stop it first.
[root@vm-CentOS75-0-149 hadoop]# /opt/bdp/hdfs/bin/hdfs haadmin -getServiceState nn1
standby
[root@vm-CentOS75-0-149 hadoop]# /opt/bdp/hdfs/bin/hdfs haadmin -getServiceState nn2
standby
[root@vm-CentOS75-0-149 hadoop]# /opt/bdp/hdfs/bin/hdfs haadmin -getServiceState nn3
active