Kubernetes ResourceQuota Controller内部实现原理及源码分析

本文是对基于Kubernetes V1.5的代码，对ResourceQuotaController的原理分析和源码分析，给出了对应的源码目录结构分析，内部实现原理图，及其完整流程的源码分析，希望能帮助你对Kubernetes ResourceQuota和ResourceQuotaController有更深入的了解。

ResoureQuota介绍

关于ResoureQuota和ResourceController的介绍和使用请参见如下官方文档。这是你理解这篇博客的基础。

• https://kubernetes.io/docs/admin/resourcequota/
• https://kubernetes.io/docs/admin/resourcequota/walkthrough/
  
• https://kubernetes.io/docs/user-guide/compute-resources/
• https://kubernetes.io/docs/admin/admission-controllers/
• https://github.com/kubernetes/community/blob/master/contributors/design-proposals/admission_control_resource_quota.md

ResourceQuota Controller源码目录结构分析

ResourceQuota Controller作为Kubernetes Controller Manager管理的众多Controller中的一员，其主要的源码位于目录k8s.io/kubernetes/pkg/quota和k8s.io/kubernetes/pkg/controller/resourcequota，具体分析如下：

k8s.io/kubernetes/pkg/quota
.
├── evaluator    // 负责各种资源使用的统计
│   └── core
│       ├── configmap.go   // ConfigMapEvaluator的实现，负责ConfigMap资源的统计
│       ├── doc.go
│       ├── persistent_volume_claims.go    // PVCEvaluator的实现，负责PVC资源的统计
│       ├── persistent_volume_claims_test.go
│       ├── pods.go    //PodEvaluator的实现，负责Pod资源的统计
│       ├── pods_test.go
│       ├── registry.go    // 创建Registry时注册所有的Evaluators
│       ├── replication_controllers.go    // RCEvaluator的实现，负责ReplicationController资源的统计
│       ├── resource_quotas.go    // ResourceQuotaEvaluator的实现，负责ResourceQuota资源的统计
│       ├── secrets.go    // SecretEvaluator的实现，负责Secret资源的统计
│       ├── services.go    // ServiceEvaluator的实现，负责Service资源的统计
│       └── services_test.go
├── generic    // genericEvaluator的定义和实现
│   ├── evaluator.go    // 实现了genericEvaluator的接口，包括最重要的CalculateUsageStats接口
│   └── registry.go    // 定义GenericRegistry
├── install
│   └── registry.go    // 定义了startResourceQuotaController时会调用创建ResourceQuota Registry的方法
├── interfaces.go    // 定义了Registry和Evaluator Interface
├── resources.go    // 定义Resources的集合操作以及CalculateUsage方法
└── resources_test.go

k8s.io/kubernetes/pkg/controller/resourcequota
.
├── doc.go
├── replenishment_controller.go    // 定义replenishmentControllerFactory，用来创建replenishmentController
├── replenishment_controller_test.go
├── resource_quota_controller.go    // 定义ResourceQuotaController及其Run方法，syncResourceQuota方法等，属于核心文件。
└── resource_quota_controller_test.go

ResourceQuota Controller内部实现原理图

请下载到本地放大查看。

具体各个模块的功能和交互请看下面的源码分析。

ResourceQuota Controller源码分析

上面的内部实现原理图显示，ResourceQuotaController是Kubenetes Controller Manager启动进行初始化众多Controllers的时候，通过调用startResourceQuotaController来完成ResourceQuotaController的启动。

从kube-controller-manager的startResourceQuotaController开始

cmd/kube-controller-manager/app/core.go:76

func startResourceQuotaController(ctx ControllerContext) (bool, error) {
    resourceQuotaControllerClient := ctx.ClientBuilder.ClientOrDie("resourcequota-controller")
    resourceQuotaRegistry := quotainstall.NewRegistry(resourceQuotaControllerClient, ctx.InformerFactory)

    // 定义ReplenishmentController需要监控的资源对象
    groupKindsToReplenish := []schema.GroupKind{
        api.Kind("Pod"),
        api.Kind("Service"),
        api.Kind("ReplicationController"),
        api.Kind("PersistentVolumeClaim"),
        api.Kind("Secret"),
        api.Kind("ConfigMap"),
    }

    ...

    go resourcequotacontroller.NewResourceQuotaController(
        resourceQuotaControllerOptions,
    ).Run(int(ctx.Options.ConcurrentResourceQuotaSyncs), ctx.Stop)
    return true, nil
}

startResourceQuotaController启动一个goroutine，通过NewResourceQuotaController创建一个ResourceQuotaController并执行其Run方法开始提供ResourceQuotaController。

下面是ResourceQuotaController和ResourceQuotaControllerOptions结构体的定义。ResourceQuotaController中定义了几个关键Entity，分别是rqController、queue、missingUsageQueue、registry、replenishmentControllers，在上一节中的原理图中也能看到它们的身影。

ResourceQuotaController定义

pkg/controller/resourcequota/resource_quota_controller.go:40

// ResourceQuotaControllerOptions holds options for creating a quota controller
type ResourceQuotaControllerOptions struct {
    // Must have authority to list all quotas, and update quota status
    KubeClient clientset.Interface
    // Controls full recalculation of quota usage
    ResyncPeriod controller.ResyncPeriodFunc
    // Knows how to calculate usage
    Registry quota.Registry
    // Knows how to build controllers that notify replenishment events
    ControllerFactory ReplenishmentControllerFactory
    // Controls full resync of objects monitored for replenihsment.
    ReplenishmentResyncPeriod controller.ResyncPeriodFunc
    // List of GroupKind objects that should be monitored for replenishment at
    // a faster frequency than the quota controller recalculation interval
    GroupKindsToReplenish []schema.GroupKind
}

// ResourceQuotaController is responsible for tracking quota usage status in the system
type ResourceQuotaController struct {
    // Must have authority to list all resources in the system, and update quota status
    kubeClient clientset.Interface
    // An index of resource quota objects by namespace
    rqIndexer cache.Indexer
    // Watches changes to all resource quota
    rqController *cache.Controller
    // ResourceQuota objects that need to be synchronized
    queue workqueue.RateLimitingInterface
    // missingUsageQueue holds objects that are missing the initial usage informatino
    missingUsageQueue workqueue.RateLimitingInterface
    // To allow injection of syncUsage for testing.
    syncHandler func(key string) error
    // function that controls full recalculation of quota usage
    resyncPeriod controller.ResyncPeriodFunc
    // knows how to calculate usage
    registry quota.Registry
    // controllers monitoring to notify for replenishment
    replenishmentControllers []cache.ControllerInterface
}

NewRegistry

接下来，我们看看startResourceQuotaController调用的NewRegistry、NewResourceQuotaController以及ResourceQuotaController的Run方法。

pkg/quota/evaluator/core/registry.go:29

// NewRegistry returns a registry that knows how to deal with