背景
作业可能会出现以下报错
ERROR org.apache.hadoop.hdfs.KeyProviderCache - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
这个报错是hdfs客户端的一个bug,但并不影响作业正常运行,且在2.8版本之后已经修复
相关代码
DistributedFileSystem.java
在这里调用getKeyProvider方法
public Token<?>[] addDelegationTokens(
final String renewer, Credentials credentials) throws IOException {
Token<?>[] tokens = super.addDelegationTokens(renewer, credentials);
// 调用dfs.getKeyProvider方法
if (dfs.getKeyProvider() != null) {
KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());
KeyProviderCache.java
最终会调用到createKeyProviderURI方法
private URI createKeyProviderURI(Configuration conf) {
final String providerUriStr =
conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, "");
// 找不到dfs.encryption.key.provider.uri配置,报错
if (providerUriStr.isEmpty()) {
LOG.error("Could not find uri with key ["
+ DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI
+ "] to create a keyProvider !!");
return null;
}
final URI providerUri;
try {
providerUri = new URI(providerUriStr);
} catch (URISyntaxException e) {
LOG.error("KeyProvider URI string is invalid [" + providerUriStr
+ "]!!", e.getCause());
return null;
}
return providerUri;
}
解决
社区已有相关patch
https://issues.apache.org/jira/browse/HDFS-7931
代码比较简单,把判断条件换成了检查配置是否存在
DFSClient.java
+ public boolean isHDFSEncryptionEnabled() {
+ return conf.get(
+ DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, null) != null;
+ }
DistributedFileSystem.java
public Token<?>[] addDelegationTokens(
final String renewer, Credentials credentials) throws IOException {
Token<?>[] tokens = super.addDelegationTokens(renewer, credentials);
- if (dfs.getKeyProvider() != null) {
+ if (dfs.isHDFSEncryptionEnabled()) {
KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension =
KeyProviderDelegationTokenExtension.
createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());