启用Kerberos的Hive操作失误解决

最新推荐文章于 2024-08-12 23:15:00 发布

邢为栋

最新推荐文章于 2024-08-12 23:15:00 发布

阅读量6.7k

点赞数 1

分类专栏： Bigdata 文章标签： hive 安全

本文链接：https://blog.csdn.net/xwd127429/article/details/106071293

版权

Bigdata 专栏收录该内容

79 篇文章 4 订阅

订阅专栏

启用了kerberos，需要使用hive主体权限进行授权，但是开始没找到hive主体的keytab，于是决定使用kadmin.local的ktadd命令为hive主体创建一个keytab，创建之后，可以通过keytab使用hive主体权限了，通过beeline连接hive，但是报错GSS initiate failed，突然心一凉，直觉告诉我，玩脱了。。。

查看hive服务，出现告警，查看metastore日志，报错如下：

2020-05-09 15:43:34,632 ERROR org.apache.thrift.transport.TSaslTransport: [pool-9-thread-105]: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
	at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) ~[?:1.8.0_252]
	at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:537) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_252]
	at javax.security.auth.Subject.doAs(Subject.java:360) [?:1.8.0_252]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) [hadoop-common-3.0.0-cdh6.3.2.jar:?]
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
	at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:858) ~[?:1.8.0_252]
	at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_252]
	at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_252]
	at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_252]
	... 14 more
Caused by: sun.security.krb5.KrbCryptoException: Checksum failed
	at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102) ~[?:1.8.0_252]
	at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_252]
	at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_252]
	at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_252]
	at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149) ~[?:1.8.0_252]
	at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:140) ~[?:1.8.0_252]
	at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_252]
	at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_252]
	at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_252]
	at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_252]
	... 14 more
Caused by: java.security.GeneralSecurityException: Checksum failed
	at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451) ~[?:1.8.0_252]
	at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272) ~[?:1.8.0_252]
	at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76) ~[?:1.8.0_252]
	at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100) ~[?:1.8.0_252]
	at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_252]
	at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_252]
	at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_252]
	at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149) ~[?:1.8.0_252]
	at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:140) ~[?:1.8.0_252]
	at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_252]
	at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_252]
	at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_252]
	at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_252]
	... 14 more
2020-05-09 15:43:34,637 ERROR org.apache.thrift.server.TThreadPoolServer: [pool-9-thread-105]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: GSS initiate failed
	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_252]
	at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_252]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) ~[hadoop-common-3.0.0-cdh6.3.2.jar:?]
	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
	at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:314) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
	... 10 more

经过观察和回顾刚才的操作，这绝对是kerberos认证出问题了，由于我重新生成了hive的凭证票据，导致凭证票据刷新，而正在运行的服务使用的还是未刷新的凭证票据，所以认证无法通过。

于是我进入Cloudera Manager控制台，关闭hive服务，对hive服务进行了重新生成keytab的操作，然后启动hive，但是问题没有解决。

经过无数次无数次无数次的生成keytab，重启服务，我突然观察到，在Cloudera Manager控制台中，hive的启动日志中(之前一直观察的是终端的metastore日志)，在执行yarn任务的时候发生了这个错误，此时我恍然大悟，这定然是hive和其他服务的kerberos认证失败导致的，于是我决定重启整个集群。

重启完整个集群后，服务终于正常了，心累。。。