【Kerberos】问题汇总 (持续更新)

1、kerberos下hdfs 的nn和jn通信问题记录

2021-12-23 17:17:21,011 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:41133:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:24,530 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:49612:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:28,145 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:43818:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:30,970 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:45254:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:33,864 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:41578:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:34,697 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:48166:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:42,884 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:52657:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:47,341 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:52387:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:49,492 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:40828:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:54,064 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:60958:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:55,070 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:34054:null (GSS initiate failed) with true cause: (GSS initiate failed)
2021-12-23 17:17:58,627 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.16.82.190:50173:null (GSS initiate failed) with true cause: (GSS initiate failed)

2021-12-23 17:30:09,469 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Using keytab /data/kerberos/hdfs.keytab, for principal HTTP/172-16-82-160@DTSTACK.COM
2021-12-23 17:30:09,473 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Using keytab /data/kerberos/hdfs.keytab, for principal HTTP/172-16-82-160@DTSTACK.COM
2021-12-23 17:30:09,501 INFO org.mortbay.log: Started SslSelectChannelConnectorSecure@172-16-82-160:60070
2021-12-23 17:30:09,521 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories!
2021-12-23 17:30:09,555 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Edit logging is async:true
2021-12-23 17:30:09,565 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: KeyProvider: null
2021-12-23 17:30:09,566 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsLock is fair: true
2021-12-23 17:30:09,567 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2021-12-23 17:30:09,598 ERROR org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: error reading hosts files:
java.io.FileNotFoundException: null (没有那个文件或目录)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
                                                                                                                                                                                 26691,1       94%
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:635)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:696)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:906)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 3
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:196)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:885)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:738)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:316)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:635)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:696)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:906)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
Response message:
This node has namespaceId '1063507604 and clusterId 'CID-258983c5-542e-43ed-a9a5-271ef8b55b84' but the requesting node expected '789529476' and 'CID-2aab532e-ed2d-4978-b8b1-140bf1632c42'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:437)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:422)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:488)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:482)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:421)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:885)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:738)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:316)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:635)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:696)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:906)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
2021-12-23 17:24:14,640 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Error replaying edit log at offset 0.  Expected transaction ID was 1
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 3
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:196)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:885)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:738)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:316)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:635)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:696)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:906)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
2021-12-23 17:24:14,742 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0.  Expected transaction ID was 1
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:203)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:885)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:738)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:316)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:635)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:696)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:906)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 3
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:196)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        ... 12 more
2021-12-23 17:24:14,759 INFO org.mortbay.log: Stopped SslSelectChannelConnectorSecure@172-16-82-160:60070
2021-12-23 17:24:14,860 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2021-12-23 17:24:14,860 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2021-12-23 17:24:14,861 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2021-12-23 17:24:14,861 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0.  Expected transaction ID was 1
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:203)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:885)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:738)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:316)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1045)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:707)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:635)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:696)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:906)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 3
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:196)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        ... 12 more
2021-12-23 17:24:14,862 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2021-12-23 17:24:14,863 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at 172-16-82-160/172.16.82.160

这里有点日志信息没放出来,就是说GSS认证失败。内网应该不会是网络问题。可能是别的证书问题导致,现在先记录下暂时没解决,现有的解决办法就是手动同步源数据信息到standby节点然后把节点恢复,我想的是能否手动触发checkpoint,standby就回去拉取edit然后合并成fsimage。这样不知道能不能。先记录在这了。

相关:
这个就是namenode和journalnode通信问题。可以重新导出keytab文件尝试下吧。
在这里插入图片描述
我的原因是导出keytab的时候一个同样的principal被我导出了两次,而且-norandkey参数没加对导致的。但是貌似还是会有上面的这个通信问题,但是貌似不影响正常使用。。暂时就这样吧,哪位知道的话可以指点下。

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值