aws eks 集群访问ecr仓库拉取镜像的认证逻辑

本文主要讨论三个问题

• ecr帮助程序在docker上如何配置
• eks集群访问ecr仓库的逻辑
• kubelet授权ecr的源码分析

ecr帮助程序

在docker环境下，可以通过在$HOME/.docker/config.json中指定凭证管理程序

docker login

aws同样提供了证书助手，避免手动执行ecr认证命令

amazon-ecr-credential-helpe

帮助程序本身是一个二进制命令，只需要在PATH中找到就可以

eks集群访问ecr仓库

在1.24集群后默认运行时成为containerd，在我们启动集群后，会发现对于一些私有的ecr仓库，即使仓库本身没有配置放行策略，节点仍旧能够从此仓库拉取镜像。

例如以下镜像托管在aws官方的ecr仓库中。然而在节点的$HOME/.docker/config.json文件中并没有发现帮助程序的配置

918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon-k8s-cni:v1.11.4-eksbuild.1

原来，从 Kubernetes v1.20 开始，kubelet 可以使用 exec 插件动态获得针对某容器镜像库的凭据。kubelet 需要设置以下两个标志：

• --image-credential-provider-config —— 凭据提供程序插件配置文件的路径。
• --image-credential-provider-bin-dir —— 凭据提供程序插件二进制可执行文件所在目录的路径。

kubelet启动节点时kubelet日志中有以下参数配置

I0218 05:30:51.783345    5014 flags.go:64] FLAG: --image-credential-provider-bin-dir="/etc/eks/ecr-credential-provider"
I0218 05:30:51.783350    5014 flags.go:64] FLAG: --image-credential-provider-config="/etc/eks/ecr-credential-provider/ecr-credential-provider-config"

去除访问ecr权限后报错如下

Warning  Failed 11s  kubelet    Failed to pull image "xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazonlinux:latest": rpc error: code = Unknown desc = failed to pull and unpack image "xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazonlinux:latest": failed to resolve reference "xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazonlinux:latest": pulling from host xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn failed with status code [manifests latst[]: 401 Unauthorized       
Warning  Failed 11s  kubelet    Error: ErrImagePull

节点上的kubelet日志显示

E0218 05:50:49.260465    5014 aws_credentials.go:184] error getting credentials from ECR for xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn AccessDeniedException: User: arn:aws-cn:sts::xxxxxx:assumed-role/eksctl-test124-nodegroup-test124-NodeInstanceRole-1NVLMK3YZWYY3/i-0a598be817afd520b is not authorized to perform: ecr:GetAuthorizationToken on resource: * because no identity-based policy allows the ecr:GetAuthorizationToken action

对应源码如下，可见kubelet通过插件向ecr请求凭证

cfg, err := p.getFromECR(parsed)
if err != nil {
    klog.Errorf("error getting credentials from ECR for %s %v", parsed.registry, err)
    return credentialprovider.DockerConfig{}
}

kubelet的凭证获取程序的配置文件，满足以下条件时匹配

• 两者都包含相同数量的域部分，并且每个部分都匹配。
• 匹配image的 URL 路径必须是目标image URL 路径的前缀。
• 如果 matchImages 包含端口，则该端口也必须在镜像中匹配。

配置 kubelet 镜像凭据提供程序

$ cat /etc/eks/ecr-credential-provider/ecr-credential-provider-config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: CredentialProviderConfig
providers:
  - name: ecr-credential-provider
    matchImages:
      - "*.dkr.ecr.*.amazonaws.com"
      - "*.dkr.ecr.*.amazonaws.cn"
      - "*.dkr.ecr-fips.*.amazonaws.com"
      - "*.dkr.ecr.us-iso-east-1.c2s.ic.gov"
      - "*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov"
    defaultCacheDuration: "12h"
    apiVersion: credentialprovider.kubelet.k8s.io/v1beta1
    args:
      - get-credentials

根据官网的说明

https://kubernetes.github.io/cloud-provider-aws/credential_provider/
https://github.com/kubernetes/enhancements/tree/master/keps/sig-cloud-provider/2133-out-of-tree-credential-provider

Once you pass this config to the kubelet, everytime it needs to fetch an image that matches one of the “matchImages” patterns, it will invoke the “ecr-credential-provider” binary in the --image-credential-provider-bin-dir folder. In turn, the plugin will fetch the credentials for kubelet and send it back via stdio.

手动调用的ecr-credential-provider的方法如下

echo '{"kind":"CredentialProviderRequest","apiVersion":"credentialprovider.kubelet.k8s.io/v1","image":"112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.12"}' | /etc/eks/image-credential-provider/ecr-credential-provider

分析源码

https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/aws/aws_credentials.go

https://github1s.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/aws/aws_credentials.go#L187

这里额外提一下github1s项目，在vscode终端中浏览github仓库，提供高亮和折叠等功能，这样就不用下载源码了

首先初始化凭证提供程序，创建一个ecr token的缓存

// init registers a credential provider for each registryURLTemplate and creates
// an ECR token getter factory with a new cache to store token getters
func init() {
	credentialprovider.RegisterCredentialProvider("amazon-ecr",
		newECRProvider(&ecrTokenGetterFactory{cache: make(map[string]tokenGetter)},
			ec2ValidationImpl,
		))
}

以上init函数初始化了一个包含ecrProvider的对象，包括缓存，token工程函数，一个判断ec2环境的函数

type ecrProvider struct {
	cache         cache.Store
	getterFactory tokenGetterFactory
	isEC2         ec2ValidationFunc
}

核心逻辑是一个名为Provider的方法

func (p *ecrProvider) Provide(image string) credentialprovider.DockerConfig {
	parsed, err := parseRepoURL(image)
	if err != nil {
		return credentialprovider.DockerConfig{}
	}
    
    // 避免由于非aws平台造成的aws sdk执行延迟，判断是否为ec2环境
    // 具体方法是ecrProvider中的isEC2函数，依据为 1. 获取实例的uuid 2. 获取凭证session
    // 只会执行一次
    once.Do(func() {
		isEC2 = p.isEC2()

		if isEC2 && credentialprovider.AreLegacyCloudCredentialProvidersDisabled() {
			klog.V(4).Infof("AWS credential provider is now disabled. Please refer to sig-cloud-provider for guidance on external credential provider integration for AWS")
		}
	})
    
    // 从cache中查找ecr token
    if cfg, exists := p.getFromCache(parsed); exists {
		klog.V(3).Infof("Got ECR credentials from cache for %s", parsed.registry)
		return cfg
	}
	klog.V(3).Info("unable to get ECR credentials from cache, checking ECR API")
    
    // 向ecr发起获取token请求
    cfg, err := p.getFromECR(parsed)
	if err != nil {
		klog.Errorf("error getting credentials from ECR for %s %v", parsed.registry, err)
		return credentialprovider.DockerConfig{}
	}

对于新节点需要向ecr发起请求获取token

func (p *ecrProvider) getFromECR(parsed *parsedURL) (credentialprovider.DockerConfig, error) {
cfg := credentialprovider.DockerConfig{}
    // 解析区域
	getter, err := p.getterFactory.GetTokenGetterForRegion(parsed.region)
	if err != nil {
		return cfg, err
	}
    // 构造参数并发送请求到ecr获取token
	params := &ecr.GetAuthorizationTokenInput{RegistryIds: []*string{aws.String(parsed.registryID)}}
	output, err := getter.GetAuthorizationToken(params)
	...
	data := output.AuthorizationData[0]
	if data.AuthorizationToken == nil {
		return cfg, errors.New("authorization token in response is nil")
	}
    // 加入缓存
	entry, err := makeCacheEntry(data, parsed.registry)
	cfg[entry.registry] = entry.credentials
	return cfg, nil
}

前面kubelet中的错误就是由于getFromECR执行失败，导致kubelet输出error getting credentials from ECR