hadoop3高可用集群搭建异常(Can‘t format the storage directory because the current directory is not empty)

集群环境

4台虚拟机: centos7

hadoop3.3.0

jdk-8u131

zookeeper: 3.6.2

1.hdfs --daemon start journalnode

node01, node02,node03执行正确。为什么node01, node02,node03是因为把这三个节点作为journalnode。

2.node01执行hdfs namenode -format异常如下:

org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 1 exceptions thrown:
192.168.2.104:8485: Directory /var/sxt/hadoop/ha/journalnode/mycluster is in an inconsistent state: Can't format the storage directory because the current directory is not empty.
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:622)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:706)
        at org.apache.hadoop.hdfs.qjournal.server.JNStorage.format(JNStorage.java:219)
        at org.apache.hadoop.hdfs.qjournal.server.Journal.format(Journal.java:257)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.format(JournalNodeRpcServer.java:183)
        at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.format(QJournalProtocolServerSideTranslatorPB.java:150)
        at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:31922)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

        at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:305)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.format(QuorumJournalManager.java:264)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.formatNonFileJournals(FSEditLog.java:429)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:189)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1271)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1713)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1821)
2020-12-29 15:07:02,479 ERROR namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 1 exceptions thrown:
192.168.2.104:8485: Directory /var/sxt/hadoop/ha/journalnode/mycluster is in an inconsistent state: Can't format the storage directory because the current directory is not empty.
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:622)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:706)
        at org.apache.hadoop.hdfs.qjournal.server.JNStorage.format(JNStorage.java:219)
        at org.apache.hadoop.hdfs.qjournal.server.Journal.format(Journal.java:257)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.format(JournalNodeRpcServer.java:183)
        at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.format(QJournalProtocolServerSideTranslatorPB.java:150)
        at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:31922)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

        at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:305)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.format(QuorumJournalManager.java:264)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.formatNonFileJournals(FSEditLog.java:429)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:189)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1271)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1713)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1821)
2020-12-29 15:07:02,487 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 1 exceptions thrown:
192.168.2.104:8485: Directory /var/sxt/hadoop/ha/journalnode/mycluster is in an inconsistent state: Can't format the storage directory because the current directory is not empty.
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:622)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:706)
        at org.apache.hadoop.hdfs.qjournal.server.JNStorage.format(JNStorage.java:219)
        at org.apache.hadoop.hdfs.qjournal.server.Journal.format(Journal.java:257)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.format(JournalNodeRpcServer.java:183)
        at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.format(QJournalProtocolServerSideTranslatorPB.java:150)
        at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:31922)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

2020-12-29 15:07:02,492 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node01/192.168.2.104
************************************************************/
[root@node01 hadoop]# 

解决方案,经过排查发现,hdfs-site.xml配置错误导致,错误配置如下:

dfs.namenode.shared.edits.dir qjournal://node01:8485; node02:8485; node02:8485/mycluster
<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value>
</property>

修改配置后,在node01, node02,node03执行hdfs --daemon stop journalnode停止journalnode

在这三个节点再次执行hdfs --daemon start journalnode,在执行hdfs namenode -format不在报错,成功格式化。

  • 0
    点赞
  • 4
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值