hadoop kerberos java客户端报错

背景

使用hadoop java客户端访问带有kerberos认证的hadoop集群时，出现了Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]的错误。

输出日志

[WARN ] 2020-12-15 15:04:35,904  org.apache.hadoop.util.NativeCodeLoader(NativeCodeLoader.java:62) -104  [main]  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[INFO ] 2020-12-15 15:04:36,181  org.apache.hadoop.security.UserGroupInformation(UserGroupInformation.java:965) -381  [main]  - Login successful for user hdfs@HADOOPKDC using keytab file /home/config/hdfs.headless.keytab
[WARN ] 2020-12-15 15:04:36,672  org.apache.hadoop.ipc.Client(Client.java:677) -872  [main]  - Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "10.10.10.10"; destination host is: "10.10.10.11":8020; 
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
        at org.apache.hadoop.ipc.Client.call(Client.java:1480)
        at org.apache.hadoop.ipc.Client.call(Client.java:1407)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy9.getListing(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy10.getListing(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2091)
        at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2074)
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:791)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
        at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
        at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
        at com.upupfeng.kerberos.HasKerberos.main(HasKerberos.java:23)
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:732)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529)
        at org.apache.hadoop.ipc.Client.call(Client.java:1446)
        ... 20 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:720)
        ... 23 more

分析

上面的日志中有一句Login successful for user hdfs@HADOOPKDC using keytab file /home/config/hdfs.headless.keytab可以说明它是已经认证成功过一次了，只不过是某种原因导致报错了。这句就可以刷掉网上80%的文章了。

在经过一番掉头发之后，在茫茫的文章了找到了两篇文章，并解决了问题。文章链接贴在文末了。

上面的日志级别是info，看到的日志不是很全，所以让我一开始搜到的都是无关的东西。将日志级别设置为debug，就可以看出问题了。贴出两句关键debug日志：

SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:null

SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:null

通过这两句报错搜索就可以搜到关键答案了。

根据搜到的两篇文章，简单说明一下问题出现的原因。

导致这个问题出现的原因:

打包时使用maven-shade-plugin将依赖一并打包时，会导致META-INF/services/中的org.apache.hadoop.security.AnnotatedSecurityInfo被漏掉。

由于AnnotatedSecurityInfo被漏掉，导致了SecurityInfo实例不能被正常加载，就导致了最终的报错。

解决办法

解决办法1

手动设置SecurityInfo实例。

在配置kerberos代码的前面添加如下代码，手动指定SecurityInfo的实例，下面用的是AnnotatedSecurityInfo

SecurityUtil.setSecurityInfoProviders(new AnnotatedSecurityInfo());

解决办法2

通过普通的打包方式，将应用程序和依赖jar包分开打包。

运行时通过添加classpath添加进来。

占位

在这里贴上报错信息，提高搜索触达率，希望帮到更多的朋友。

• SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:null
• SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:null
• Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
• java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: ; destination host is: ;

具体分析可参考以下两篇文章

参考

https://funclojure.tumblr.com/post/155129283948/hdfs-kerberos-java-client-api-pains

https://arch-long.cn/articles/hadoop/Kerberos-Exceptions.html