kube-controller-manager是如何加载的Deployment、StatefulSet等控制器的？

假装这里是StackOverflow

已于 2023-07-04 19:28:21 修改

阅读量141

点赞数

分类专栏： k8s 文章标签： kubernetes 容器云原生

于 2023-06-28 20:05:20 首次发布

本文链接：https://blog.csdn.net/qq_21365961/article/details/131437148

版权

k8s 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

背景

我们知道，k8s的master分为三个部分：

ApiServer。负责对外提供Api接口，以及etcd数据的读写
Controller-mamager。负责k8s默认支持的workload控制器
Scheduler。负责调度Pod到Node节点上

既然Controller-manager里内置了许许多多k8s工作负载的控制器，它在哪里加载这些控制器的呢？

问题的产生

我们看到k8s项目的源代码，pkg路径下controller包有如下30个控制器

tree ./pkg/controller -d -L 1 

./pkg/controller
├── apis
├── bootstrap
├── certificates
├── clusterroleaggregation
├── cronjob
├── daemon
├── deployment
├── disruption
├── endpoint
├── endpointslice
├── endpointslicemirroring
├── garbagecollector
├── history
├── job
├── namespace
├── nodeipam
├── nodelifecycle
├── podautoscaler
├── podgc
├── replicaset
├── replication
├── resourcequota
├── serviceaccount
├── statefulset
├── storageversiongc
├── testutil
├── ttl
├── ttlafterfinished
├── util
└── volume

31 directories

再看到controller-manager的代码，这里是我们搭建一个k8s集群运行的kube-controller-manager二进制文件生成的地方

tree ./cmd/kube-controller-manager -L 1

./cmd/kube-controller-manager
├── OWNERS
├── app
└── controller-manager.go

2 directories, 2 files

此时疑问就产生了，/cmd/kube-controller-manager里controller-manager是如何跟/pkg/controller里这么多controller建立起联系的呢？
pkg里这么多controller，controller-manager不应该一个一个加载吧？

代码分析

controller-manager入口

从controller-manager的入口处开始分析

cat ./cmd/kube-controller-manager/controller-manager.go

可以看到该文件的内容十分简洁，关键的内容在app.NewControllerManagerCommand()这个方法里，由它负责关联，初始化好的command执行Execute()时，会运行起一个一个Controller。

package main

import (
	"math/rand"
	"os"
	"time"

	"github.com/spf13/pflag"

	cliflag "k8s.io/component-base/cli/flag"
	"k8s.io/component-base/logs"
	_ "k8s.io/component-base/logs/json/register"          // for JSON log format registration
	_ "k8s.io/component-base/metrics/prometheus/clientgo" // load all the prometheus client-go plugin
	_ "k8s.io/component-base/metrics/prometheus/version"  // for version metric registration
	"k8s.io/kubernetes/cmd/kube-controller-manager/app"
)

func main() {
	rand.Seed(time.Now().UnixNano())

	pflag.CommandLine.SetNormalizeFunc(cliflag.WordSepNormalizeFunc)

	command := app.NewControllerManagerCommand() // 关键内容

	logs.InitLogs()
	defer logs.FlushLogs()

	if err := command.Execute(); err != nil {
		os.Exit(1)
	}
}

进入到NewControllerManagerCommand()方法内

import (
	...//省略不重要的包
	"k8s.io/kubernetes/cmd/kube-controller-manager/app/options"
	...//省略不重要的包
)

// NewControllerManagerCommand creates a *cobra.Command object with default parameters
func NewControllerManagerCommand() *cobra.Command {
	s, err := options.NewKubeControllerManagerOptions()	//关键代码
	if err != nil {
		klog.Fatalf("unable to initialize command options: %v", err)
	}

	cmd := &cobra.Command{...} // 这部分代码省略了

	fs := cmd.Flags()
	namedFlagSets := s.Flags(KnownControllers(), ControllersDisabledByDefault.List())
	verflag.AddFlags(namedFlagSets.FlagSet("global"))
	globalflag.AddGlobalFlags(namedFlagSets.FlagSet("global"), cmd.Name())
	registerLegacyGlobalFlags(namedFlagSets)
	for _, f := range namedFlagSets.FlagSets {
		fs.AddFlagSet(f)
	}

	cols, _, _ := term.TerminalSize(cmd.OutOrStdout())
	cliflag.SetUsageAndHelpFunc(cmd, namedFlagSets, cols)

	return cmd
}

进入到options.NewKubeControllerManagerOptions()方法内，在其内部初始化了众多controller

func NewKubeControllerManagerOptions() (*KubeControllerManagerOptions, error) {
	componentConfig, err := NewDefaultComponentConfig()
	if err != nil {
		return nil, err
	}

	s := KubeControllerManagerOptions{
		Generic:         cmoptions.NewGenericControllerManagerConfigurationOptions(&componentConfig.Generic),
		KubeCloudShared: cpoptions.NewKubeCloudSharedOptions(&componentConfig.KubeCloudShared),
		ServiceController: &cpoptions.ServiceControllerOptions{
			ServiceControllerConfiguration: &componentConfig.ServiceController,
		},
		AttachDetachController: &AttachDetachControllerOptions{
			&componentConfig.AttachDetachController,
		},
		CSRSigningController: &CSRSigningControllerOptions{
			&componentConfig.CSRSigningController,
		},
		DaemonSetController: &DaemonSetControllerOptions{
			&componentConfig.DaemonSetController,
		},
		DeploymentController: &DeploymentControllerOptions{
			&componentConfig.DeploymentController,
		},
		StatefulSetController: &StatefulSetControllerOptions{
			&componentConfig.StatefulSetController,
		},
		DeprecatedFlags: &DeprecatedControllerOptions{
			&componentConfig.DeprecatedController,
		},
		EndpointController: &EndpointControllerOptions{
			&componentConfig.EndpointController,
		},
		EndpointSliceController: &EndpointSliceControllerOptions{
			&componentConfig.EndpointSliceController,
		},
		EndpointSliceMirroringController: &EndpointSliceMirroringControllerOptions{
			&componentConfig.EndpointSliceMirroringController,
		},
		GarbageCollectorController: &GarbageCollectorControllerOptions{
			&componentConfig.GarbageCollectorController,
		},
		HPAController: &HPAControllerOptions{
			&componentConfig.HPAController,
		},
		JobController: &JobControllerOptions{
			&componentConfig.JobController,
		},
		CronJobController: &CronJobControllerOptions{
			&componentConfig.CronJobController,
		},
		NamespaceController: &NamespaceControllerOptions{
			&componentConfig.NamespaceController,
		},
		NodeIPAMController: &NodeIPAMControllerOptions{
			&componentConfig.NodeIPAMController,
		},
		NodeLifecycleController: &NodeLifecycleControllerOptions{
			&componentConfig.NodeLifecycleController,
		},
		PersistentVolumeBinderController: &PersistentVolumeBinderControllerOptions{
			&componentConfig.PersistentVolumeBinderController,
		},
		PodGCController: &PodGCControllerOptions{
			&componentConfig.PodGCController,
		},
		ReplicaSetController: &ReplicaSetControllerOptions{
			&componentConfig.ReplicaSetController,
		},
		ReplicationController: &ReplicationControllerOptions{
			&componentConfig.ReplicationController,
		},
		ResourceQuotaController: &ResourceQuotaControllerOptions{
			&componentConfig.ResourceQuotaController,
		},
		SAController: &SAControllerOptions{
			&componentConfig.SAController,
		},
		TTLAfterFinishedController: &TTLAfterFinishedControllerOptions{
			&componentConfig.TTLAfterFinishedController,
		},
		SecureServing:  apiserveroptions.NewSecureServingOptions().WithLoopback(),
		Authentication: apiserveroptions.NewDelegatingAuthenticationOptions(),
		Authorization:  apiserveroptions.NewDelegatingAuthorizationOptions(),
		Metrics:        metrics.NewOptions(),
		Logs:           logs.NewOptions(),
	}
... //省略了很多代码
}

从中可以发现，大量的Controller都是由componentConfig赋值的，因此聚焦到componentConfig产生的源头，本方法的第一行—— NewDefaultComponentConfig()

import (
	...//省略不重要的包
	kubectrlmgrconfigv1alpha1 "k8s.io/kube-controller-manager/config/v1alpha1"
	kubecontrollerconfig "k8s.io/kubernetes/cmd/kube-controller-manager/app/config"
	kubectrlmgrconfig "k8s.io/kubernetes/pkg/controller/apis/config"
	kubectrlmgrconfigscheme "k8s.io/kubernetes/pkg/controller/apis/config/scheme"
	...//省略不重要的包
)

// NewDefaultComponentConfig returns kube-controller manager configuration object.
func NewDefaultComponentConfig() (kubectrlmgrconfig.KubeControllerManagerConfiguration, error) {
	versioned := kubectrlmgrconfigv1alpha1.KubeControllerManagerConfiguration{}
	kubectrlmgrconfigscheme.Scheme.Default(&versioned)

	internal := kubectrlmgrconfig.KubeControllerManagerConfiguration{}
	if err := kubectrlmgrconfigscheme.Scheme.Convert(&versioned, &internal, nil); err != nil {
		return internal, err
	}
	return internal, nil
}

从这段代码可以看到：

该方法最终返回的internal变量，就是上一级去给各个控制器赋值的componentConfig
而internal的产生，又和versioned密不可分，通过kubectrlmgrconfigscheme.Scheme.Convert(&versioned, &internal, nil)函数名语义上理解，internal是由versioned转换而成的。
versioned初始化时只是一个空的KubeControllerManagerConfiguration结构体，真正给他赋值的还要看Default方法

让我们深入探索一下kubectrlmgrconfigscheme.Scheme.Default()方法。从import的包名可以看出，kubectrlmgrconfigscheme来源于/pkg/controller/，这里终于和我们一开始提出的问题相关联了——/cmd/kube-controller-manager里controller-manager是如何跟/pkg/controller里这么多controller建立起联系的呢？

Scheme

Scheme定义的位置在"k8s.io/apimachinery/pkg/runtime/"下。

// Default sets defaults on the provided Object.
func (s *Scheme) Default(src Object) {
	if fn, ok := s.defaulterFuncs[reflect.TypeOf(src)]; ok {
		fn(src)
	}
}

kubectrlmgrconfigscheme.Scheme.Default()方法非常简单，用一个key为类型，value为处理函数的map，来处理任意有映射关系的对象。所以我们的问题变成：

这个类型与处理函数是如何初始化的，即defaulterFuncs是如何初始化的？kubectrlmgrconfigv1alpha1.KubeControllerManagerConfiguration{}这个类型对应的处理函数是什么？

答案就在Default方法的上面，另一个Scheme的类方法AddTypeDefaultingFunc()，通过注释得知，AddTypeDefaultingFunc 注册一个函数，该函数传递指向对象的指针，并且可以初始化对象上的字段。调用 Default（） 时将调用这些函数。除非默认对象与 srcType 匹配，否则永远不会调用该函数。如果使用相同的 srcType 调用此AddTypeDefaultingFunc两次，后来的会覆盖之前的。

// AddTypeDefaultingFunc registers a function that is passed a pointer to an
// object and can default fields on the object. These functions will be invoked
// when Default() is called. The function will never be called unless the
// defaulted object matches srcType. If this function is invoked twice with the
// same srcType, the fn passed to the later call will be used instead.
func (s *Scheme) AddTypeDefaultingFunc(srcType Object, fn func(interface{})) {
	s.defaulterFuncs[reflect.TypeOf(srcType)] = fn
}

于是，我们查看AddTypeDefaultingFunc的调用，相当的多，到处都有调用该方法注册初始化函数的地方。我们的目的是找到kubectrlmgrconfigv1alpha1.KubeControllerManagerConfiguration{}这个类型对应的处理函数是什么
在代码里搜索AddTypeDefaultingFunc

v1alpha1

看到上面NewDefaultComponentConfig里kubectrlmgrconfigv1alpha1.KubeControllerManagerConfiguration{}这个类型的包实际上是kube-controller-manager/config/v1alpha1

import (
	...//省略不重要的包
	kubectrlmgrconfigv1alpha1 "k8s.io/kube-controller-manager/config/v1alpha1"
	...//省略不重要的包
	)

所以我们定位到kube-controller-manager/config/v1alpha1里调用AddTypeDefaultingFunc的地方：
pkg/controller/apis/config/v1alpha1/zz_generated.defaults.go:32

// RegisterDefaults adds defaulters functions to the given scheme.
// Public to allow building arbitrary schemes.
// All generated defaulters are covering - they call all nested defaulters.
func RegisterDefaults(scheme *runtime.Scheme) error {
	scheme.AddTypeDefaultingFunc(&v1alpha1.KubeControllerManagerConfiguration{}, func(obj interface{}) {
		SetObjectDefaults_KubeControllerManagerConfiguration(obj.(*v1alpha1.KubeControllerManagerConfiguration))
	})
	return nil
}

func SetObjectDefaults_KubeControllerManagerConfiguration(in *v1alpha1.KubeControllerManagerConfiguration) {
	SetDefaults_KubeControllerManagerConfiguration(in)
	cloudproviderconfigv1alpha1.SetDefaults_KubeCloudSharedConfiguration(&in.KubeCloudShared)
}

秘密隐藏在SetDefaults_KubeControllerManagerConfiguration这个方法内，预测这个方法内会加载一系列的控制器了


func SetDefaults_KubeControllerManagerConfiguration(obj *kubectrlmgrconfigv1alpha1.KubeControllerManagerConfiguration) {
	if obj.DeprecatedController.RegisterRetryCount == 0 {
		obj.DeprecatedController.RegisterRetryCount = 10
	}

	// These defaults override the recommended defaults from the componentbaseconfigv1alpha1 package that are applied automatically
	// These client-connection defaults are specific to the kube-controller-manager
	if obj.Generic.ClientConnection.QPS == 0.0 {
		obj.Generic.ClientConnection.QPS = 20.0
	}
	if obj.Generic.ClientConnection.Burst == 0 {
		obj.Generic.ClientConnection.Burst = 30
	}

	// Use the default RecommendedDefaultGenericControllerManagerConfiguration options
	cmconfigv1alpha1.RecommendedDefaultGenericControllerManagerConfiguration(&obj.Generic)
	// Use the default RecommendedDefaultHPAControllerConfiguration options
	attachdetachconfigv1alpha1.RecommendedDefaultAttachDetachControllerConfiguration(&obj.AttachDetachController)
	// Use the default RecommendedDefaultCSRSigningControllerConfiguration options
	csrsigningconfigv1alpha1.RecommendedDefaultCSRSigningControllerConfiguration(&obj.CSRSigningController)
	// Use the default RecommendedDefaultDaemonSetControllerConfiguration options
	daemonconfigv1alpha1.RecommendedDefaultDaemonSetControllerConfiguration(&obj.DaemonSetController)
	// Use the default RecommendedDefaultDeploymentControllerConfiguration options
	deploymentconfigv1alpha1.RecommendedDefaultDeploymentControllerConfiguration(&obj.DeploymentController)
	// Use the default RecommendedDefaultStatefulSetControllerConfiguration options
	statefulsetconfigv1alpha1.RecommendedDefaultStatefulSetControllerConfiguration(&obj.StatefulSetController)
	// Use the default RecommendedDefaultEndpointControllerConfiguration options
	endpointconfigv1alpha1.RecommendedDefaultEndpointControllerConfiguration(&obj.EndpointController)
	// Use the default RecommendedDefaultEndpointSliceControllerConfiguration options
	endpointsliceconfigv1alpha1.RecommendedDefaultEndpointSliceControllerConfiguration(&obj.EndpointSliceController)
	// Use the default RecommendedDefaultEndpointSliceMirroringControllerConfiguration options
	endpointslicemirroringconfigv1alpha1.RecommendedDefaultEndpointSliceMirroringControllerConfiguration(&obj.EndpointSliceMirroringController)
	// Use the default RecommendedDefaultGenericControllerManagerConfiguration options
	garbagecollectorconfigv1alpha1.RecommendedDefaultGarbageCollectorControllerConfiguration(&obj.GarbageCollectorController)
	// Use the default RecommendedDefaultJobControllerConfiguration options
	jobconfigv1alpha1.RecommendedDefaultJobControllerConfiguration(&obj.JobController)
	// Use the default RecommendedDefaultCronJobControllerConfiguration options
	cronjobconfigv1alpha1.RecommendedDefaultCronJobControllerConfiguration(&obj.CronJobController)
	// Use the default RecommendedDefaultNamespaceControllerConfiguration options
	namespaceconfigv1alpha1.RecommendedDefaultNamespaceControllerConfiguration(&obj.NamespaceController)
	// Use the default RecommendedDefaultNodeIPAMControllerConfiguration options
	nodeipamconfigv1alpha1.RecommendedDefaultNodeIPAMControllerConfiguration(&obj.NodeIPAMController)
	// Use the default RecommendedDefaultHPAControllerConfiguration options
	poautosclerconfigv1alpha1.RecommendedDefaultHPAControllerConfiguration(&obj.HPAController)
	// Use the default RecommendedDefaultNodeLifecycleControllerConfiguration options
	nodelifecycleconfigv1alpha1.RecommendedDefaultNodeLifecycleControllerConfiguration(&obj.NodeLifecycleController)
	// Use the default RecommendedDefaultPodGCControllerConfiguration options
	podgcconfigv1alpha1.RecommendedDefaultPodGCControllerConfiguration(&obj.PodGCController)
	// Use the default RecommendedDefaultReplicaSetControllerConfiguration options
	replicasetconfigv1alpha1.RecommendedDefaultReplicaSetControllerConfiguration(&obj.ReplicaSetController)
	// Use the default RecommendedDefaultReplicationControllerConfiguration options
	replicationconfigv1alpha1.RecommendedDefaultReplicationControllerConfiguration(&obj.ReplicationController)
	// Use the default RecommendedDefaultResourceQuotaControllerConfiguration options
	resourcequotaconfigv1alpha1.RecommendedDefaultResourceQuotaControllerConfiguration(&obj.ResourceQuotaController)
	// Use the default RecommendedDefaultGenericControllerManagerConfiguration options
	serviceconfigv1alpha1.RecommendedDefaultServiceControllerConfiguration(&obj.ServiceController)
	// Use the default RecommendedDefaultSAControllerConfiguration options
	serviceaccountconfigv1alpha1.RecommendedDefaultSAControllerConfiguration(&obj.SAController)
	// Use the default RecommendedDefaultTTLAfterFinishedControllerConfiguration options
	ttlafterfinishedconfigv1alpha1.RecommendedDefaultTTLAfterFinishedControllerConfiguration(&obj.TTLAfterFinishedController)
	// Use the default RecommendedDefaultPersistentVolumeBinderControllerConfiguration options
	persistentvolumeconfigv1alpha1.RecommendedDefaultPersistentVolumeBinderControllerConfiguration(&obj.PersistentVolumeBinderController)
}

果不其然，在SetDefaults_KubeControllerManagerConfiguration这个方法内，加载了一系列的控制器，以Deployment为例，Default配置里仅推荐了ConcurrentDeploymentSync和DeploymentControllerSyncPeriod的设置

// RecommendedDefaultDeploymentControllerConfiguration defaults a pointer to a
// DeploymentControllerConfiguration struct. This will set the recommended default
// values, but they may be subject to change between API versions. This function
// is intentionally not registered in the scheme as a "normal" `SetDefaults_Foo`
// function to allow consumers of this type to set whatever defaults for their
// embedded configs. Forcing consumers to use these defaults would be problematic
// as defaulting in the scheme is done as part of the conversion, and there would
// be no easy way to opt-out. Instead, if you want to use this defaulting method
// run it in your wrapper struct of this type in its `SetDefaults_` method.
func RecommendedDefaultDeploymentControllerConfiguration(obj *kubectrlmgrconfigv1alpha1.DeploymentControllerConfiguration) {
	zero := metav1.Duration{}
	if obj.ConcurrentDeploymentSyncs == 0 {
		obj.ConcurrentDeploymentSyncs = 5
	}
	if obj.DeploymentControllerSyncPeriod == zero {
		obj.DeploymentControllerSyncPeriod = metav1.Duration{Duration: 30 * time.Second}
	}
}

controllermanager

至此，拿到了各个控制器的默认配置，接下里就是通过默认配置去初始化一个个控制器了，我们回到
入口的NewControllerManagerCommand()方法内，上述隐藏了cmd的一系列初始化过程

cmd := &cobra.Command{...}

从command结构体的定义上可以看到，cobra的command在运行时，会有一个运行顺序：

	// The *Run functions are executed in the following order:
	//   * PersistentPreRun()
	//   * PreRun()
	//   * Run()
	//   * PostRun()
	//   * PersistentPostRun()
	// All functions get the same args, the arguments after the command name.
	//
	// PersistentPreRun: children of this command will inherit and execute.
	PersistentPreRun func(cmd *Command, args []string)
	// PersistentPreRunE: PersistentPreRun but returns an error.
	PersistentPreRunE func(cmd *Command, args []string) error
	// PreRun: children of this command will not inherit.
	PreRun func(cmd *Command, args []string)
	// PreRunE: PreRun but returns an error.
	PreRunE func(cmd *Command, args []string) error
	// Run: Typically the actual work function. Most commands will only implement this.
	Run func(cmd *Command, args []string)
	// RunE: Run but returns an error.
	RunE func(cmd *Command, args []string) error
	// PostRun: run after the Run command.
	PostRun func(cmd *Command, args []string)
	// PostRunE: PostRun but returns an error.
	PostRunE func(cmd *Command, args []string) error
	// PersistentPostRun: children of this command will inherit and execute after PostRun.
	PersistentPostRun func(cmd *Command, args []string)
	// PersistentPostRunE: PersistentPostRun but returns an error.
	PersistentPostRunE func(cmd *Command, args []string) error

这里我们暂时把如何执行到Run方法的过程放一放，只需要知道main函数里，command执行Execute()时会运行到Run方法，这个之后单独写一个
我们直接看到Run方法，其中隐去了很多代码，找到最关键的选主逻辑。既然controller-manager是个控制器，那它一定是单独工作的，如果一起工作会导致结果不一致，保证高可用的方式是通过选主切换来完成的。


// Run runs the KubeControllerManagerOptions.  This should never exit.
func Run(c *config.CompletedConfig, stopCh <-chan struct{}) error {

	//... 省略很多代码
	run := func(ctx context.Context, startSATokenController InitFunc, initializersFunc ControllerInitializersFunc) {

		controllerContext, err := CreateControllerContext(c, rootClientBuilder, clientBuilder, ctx.Done())
		if err != nil {
			klog.Fatalf("error building controller context: %v", err)
		}
		controllerInitializers := initializersFunc(controllerContext.LoopMode)
		if err := StartControllers(controllerContext, startSATokenController, controllerInitializers, unsecuredMux); err != nil {
			klog.Fatalf("error starting controllers: %v", err)
		}

		controllerContext.InformerFactory.Start(controllerContext.Stop)
		controllerContext.ObjectOrMetadataInformerFactory.Start(controllerContext.Stop)
		close(controllerContext.InformersStarted)

		select {}
	}
	//... 省略很多代码
	// Start the main lock
	go leaderElectAndRun(c, id, electionChecker,
		c.ComponentConfig.Generic.LeaderElection.ResourceLock,
		c.ComponentConfig.Generic.LeaderElection.ResourceName,
		leaderelection.LeaderCallbacks{
			OnStartedLeading: func(ctx context.Context) {
				//关键代码
				initializersFunc := NewControllerInitializers
				if leaderMigrator != nil {
					// If leader migration is enabled, we should start only non-migrated controllers
					//  for the main lock.
					initializersFunc = createInitializersFunc(leaderMigrator.FilterFunc, leadermigration.ControllerNonMigrated)
					klog.Info("leader migration: starting main controllers.")
				}
				//关键代码
				run(ctx, startSATokenController, initializersFunc)
			},
			OnStoppedLeading: func() {
				klog.Fatalf("leaderelection lost")
			},
		})

	//... 省略很多代码

	select {}
}

两处关键的代码，一个是加载函数执行的方法，另一个是执行这些函数执行方法，由上面定义的一个内部函数run所控制。

执行函数的关联

先看加载函数执行的方法，找到initializersFunc := NewControllerInitializers，此处同样是个一个函数map，里面加载了各种控制器的start操作：

// NewControllerInitializers is a public map of named controller groups (you can start more than one in an init func)
// paired to their InitFunc.  This allows for structured downstream composition and subdivision.
func NewControllerInitializers(loopMode ControllerLoopMode) map[string]InitFunc {
	controllers := map[string]InitFunc{}
	controllers["endpoint"] = startEndpointController
	controllers["endpointslice"] = startEndpointSliceController
	controllers["endpointslicemirroring"] = startEndpointSliceMirroringController
	controllers["replicationcontroller"] = startReplicationController
	controllers["podgc"] = startPodGCController
	controllers["resourcequota"] = startResourceQuotaController
	controllers["namespace"] = startNamespaceController
	controllers["serviceaccount"] = startServiceAccountController
	controllers["garbagecollector"] = startGarbageCollectorController
	controllers["daemonset"] = startDaemonSetController
	controllers["job"] = startJobController
	controllers["deployment"] = startDeploymentController
	controllers["replicaset"] = startReplicaSetController
	controllers["horizontalpodautoscaling"] = startHPAController
	controllers["disruption"] = startDisruptionController
	controllers["statefulset"] = startStatefulSetController
	controllers["cronjob"] = startCronJobController
	controllers["csrsigning"] = startCSRSigningController
	controllers["csrapproving"] = startCSRApprovingController
	controllers["csrcleaner"] = startCSRCleanerController
	controllers["ttl"] = startTTLController
	controllers["bootstrapsigner"] = startBootstrapSignerController
	controllers["tokencleaner"] = startTokenCleanerController
	controllers["nodeipam"] = startNodeIpamController
	controllers["nodelifecycle"] = startNodeLifecycleController
	//... 省略很多代码
	controllers["persistentvolume-binder"] = startPersistentVolumeBinderController
	controllers["attachdetach"] = startAttachDetachController
	controllers["persistentvolume-expander"] = startVolumeExpandController
	controllers["clusterrole-aggregation"] = startClusterRoleAggregrationController
	controllers["pvc-protection"] = startPVCProtectionController
	controllers["pv-protection"] = startPVProtectionController
	controllers["ttl-after-finished"] = startTTLAfterFinishedController
	controllers["root-ca-cert-publisher"] = startRootCACertPublisher
	controllers["ephemeral-volume"] = startEphemeralVolumeController
	//... 省略很多代码
	return controllers
}

还是以Deployment 为例，看看startDeploymentController都做了些什么

func startDeploymentController(ctx ControllerContext) (http.Handler, bool, error) {
	dc, err := deployment.NewDeploymentController(
		ctx.InformerFactory.Apps().V1().Deployments(),
		ctx.InformerFactory.Apps().V1().ReplicaSets(),
		ctx.InformerFactory.Core().V1().Pods(),
		ctx.ClientBuilder.ClientOrDie("deployment-controller"),
	)
	if err != nil {
		return nil, true, fmt.Errorf("error creating Deployment controller: %v", err)
	}
	go dc.Run(int(ctx.ComponentConfig.DeploymentController.ConcurrentDeploymentSyncs), ctx.Stop)
	return nil, true, nil
}

// Run begins watching and syncing.
func (dc *DeploymentController) Run(workers int, stopCh <-chan struct{}) {
	defer utilruntime.HandleCrash()
	defer dc.queue.ShutDown()

	klog.InfoS("Starting controller", "controller", "deployment")
	defer klog.InfoS("Shutting down controller", "controller", "deployment")

	if !cache.WaitForNamedCacheSync("deployment", stopCh, dc.dListerSynced, dc.rsListerSynced, dc.podListerSynced) {
		return
	}

	for i := 0; i < workers; i++ {
		go wait.Until(dc.worker, time.Second, stopCh)
	}

	<-stopCh
}

最终执行

再看执行这些函数执行方法，由这个统一的内部函数来运行每一个startController

	run := func(ctx context.Context, startSATokenController InitFunc, initializersFunc ControllerInitializersFunc) {

		controllerContext, err := CreateControllerContext(c, rootClientBuilder, clientBuilder, ctx.Done())
		if err != nil {
			klog.Fatalf("error building controller context: %v", err)
		}
		controllerInitializers := initializersFunc(controllerContext.LoopMode)
		if err := StartControllers(controllerContext, startSATokenController, controllerInitializers, unsecuredMux); err != nil {
			klog.Fatalf("error starting controllers: %v", err)
		}

		controllerContext.InformerFactory.Start(controllerContext.Stop)
		controllerContext.ObjectOrMetadataInformerFactory.Start(controllerContext.Stop)
		close(controllerContext.InformersStarted)

		select {}
	}

总结

梳理controller-manager的启动过程，回答了**/cmd/kube-controller-manager里controller-manager是如何跟/pkg/controller里这么多controller建立起联系的呢？**这个问题。总结起来分为两点：

配置的关联
运行方法的关联

我们还是回到controller-manager的入口main函数，里面有两块关键代码：

package main

import (
	"math/rand"
	"os"
	"time"

	"github.com/spf13/pflag"

	cliflag "k8s.io/component-base/cli/flag"
	"k8s.io/component-base/logs"
	_ "k8s.io/component-base/logs/json/register"          // for JSON log format registration
	_ "k8s.io/component-base/metrics/prometheus/clientgo" // load all the prometheus client-go plugin
	_ "k8s.io/component-base/metrics/prometheus/version"  // for version metric registration
	"k8s.io/kubernetes/cmd/kube-controller-manager/app"
)

func main() {
	rand.Seed(time.Now().UnixNano())

	pflag.CommandLine.SetNormalizeFunc(cliflag.WordSepNormalizeFunc)

	command := app.NewControllerManagerCommand() // 关键内容1

	logs.InitLogs()
	defer logs.FlushLogs()

	if err := command.Execute(); err != nil {	//关键内容2
		os.Exit(1)
	}
}

配置的关联

从关键内容1的梳理，我们得知，每一个controller的推荐配置，或者说默认配置，都是放在defaults.go这个文件中

比如说Deployment，defaults.go文件所在的路径为pkg/controller/deployment/config/v1alpha1/，方法名为RecommendedDefaultDeploymentControllerConfiguration
比如说StatefulSet，defaults.go文件所在的路径为pkg/controller/statefulset/config/v1alpha1/，方法名为RecommendedDefaultStatefulSetControllerConfiguration
以此类推

他们全都注册在Scheme这个中间层里的，通过pkg/apis/apps/v1beta1/zz_generated.defaults.go文件中的AddTypeDefaultingFunc方法注册到Scheme中

而/cmd/kube-controller-manager在cmd/kube-controller-manager/app/options/options.go:209中的NewDefaultComponentConfig方法，从Scheme里把所有controller的配置取出来加载。
Scheme粘合配置

因此，controller-manager和各个控制器的配置是通过Scheme进行粘合的。

运行方法的关联

从关键内容2的梳理，我们得知，controller-manager在cmd/kube-controller-manager/app/controllermanager.go:408中的NewControllerInitializers()方法，与其他各个controller相关联，注册每一个controller的构造方法

比如说Deployment，pkg/controller/deployment/deployment_controller.go:101中NewDeploymentController()方法是Deployment控制器的构造方法，该构造方法是在cmd/kube-controller-manager/app/apps.go:72中startDeploymentController进行关联的
比方说StatefulSet，pkg/controller/statefulset/stateful_set.go:80中NewStatefulSetController()方法是StatefulSet控制器的构造方法，该构造方法是在cmd/kube-controller-manager/app/apps.go:51中startStatefulSetController进行关联的
以此类推

controller-manager和各个控制器的运行方法，是直接调用进行关联的，没有其他组件参与

假装这里是StackOverflow

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
kube-controller-manager是如何加载的Deployment、StatefulSet等控制器的？

梳理controller-manager的启动过程，回答了/cmd/kube-controller-manager里controller-manager是如何跟/pkg/controller里这么多controller建立起联系的呢？这个问题。
复制链接

扫一扫