Delegation Token can be issued only with kerberos or web authentication

最新推荐文章于 2023-01-04 22:02:11 发布

wk022

最新推荐文章于 2023-01-04 22:02:11 发布

阅读量5.1k

点赞数

本文链接：https://blog.csdn.net/wk022/article/details/51208641

版权

在进行spark开发的时候遇到了这样的问题：

User class threw exception: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token can be issued only with kerberos or web authentication

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:7469)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:541)

at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getDelegationToken(AuthorizationProviderProxyClientProtocol.java:661)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:964)

at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

场景：集群使用kerberos进行了安全验证，spark应用访问HDFS，HDFS进行了HA，提交的时候采用yarn-cluster

网上搜索：

https://issues.apache.org/jira/browse/MAPREDUCE-4324

http://stackoverflow.com/questions/36224782/spark-1-5-0-and-delegation-token-can-be-issued-only-with-kerberos-or-web-authent

1，采用yarn-client，可以正确运行。

2，去掉访问HDFS的操作，采用yarn-cluster，可以正常运行， val hadoopRDD = sc.textFile( "hdfs://test01:8020/tmp/hive-site.xml" )

经过排查发现，是由于访问hdfs的地址错误，当访问HA的HDFS是应使用HA标记访问HDFS

应将

val hadoopRDD = sc.textFile( "hdfs://test01:8020/tmp/hive-site.xml" )

改为

val hadoopRDD = sc.textFile( "hdfs://HA标记/tmp/hive-site.xml" )