环境和场景
环境:CDH6.3.1+Kerberos
场景:数据经Flink处理后写入到Hbase
问题一:Zookeeper认证错误
报错内容:Flink Task Managers日志
2022-03-14 15:02:25,599 INFO org.apache.zookeeper.ClientCnxn [] - Session establishment complete on server 10.53.xx.xx/10.53.xx.xx:2181, sessionid = 0x37f7cf3df130eeb, negotiated timeout = 60000
2022-03-14 15:02:25,613 ERROR org.apache.zookeeper.client.ZooKeeperSaslClient [] - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2022-03-14 15:02:25,614 ERROR org.apache.zookeeper.ClientCnxn [] - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2022-03-14 15:02:25,719 INFO org.apache.flink.kafka.shaded.org.apache.kafka.clients.Metadata [] - [Producer clientId=producer-2] Cluster ID: N2jGIl7MRnurZjFbsk-M7g
/var/log/krb5kdc.log 日志
Mar 15 11:42:06 cdh3 krb5kdc[1600](info): AS_REQ (4 etypes {18 17 16 23}) 10.53.xx.xx: ISSUE: authtime 1647315726, etypes {rep=18 tkt=18 ses=18}, hdfs/hdfs@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Mar 15 11:42:06 cdh3 krb5kdc[1600](info): TGS_REQ (4 etypes {18 17 16 23}) 10.53.xx.xx: LOOKING_UP_SERVER: authtime 0, hdfs/hdfs@EXAMPLE.COM for zookeeper/10.53.xx.xx@EXAMPLE.COM, Server not found in Kerberos database
Mar 15 11:42:07 cdh3 krb5kdc[1600](info): AS_REQ (8 etypes {18 17 16 23 25 26 20 19}) 10.53.xx.xx: ISSUE: authtime 1647315727, etypes {rep=18 tkt=18 ses=18}, HTTP/cdh2@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
问题原因初步分析:Zookeeper的kerberos认证失败,在Kerberos databas中没有找到用户zookeeper/10.53.xx.xx@EXAMPLE.