Kubernetes kube-controller-manager 控制中心机制源码深入剖析-Kubernetes商业环境实战

最新推荐文章于 2024-04-20 11:59:30 发布

weixin_30677617

最新推荐文章于 2024-04-20 11:59:30 发布

阅读量276

点赞数

文章标签：大数据网络 runtime

原文链接：https://juejin.im/post/5d614862f265da03b31bd897

版权

专注于大数据及容器云核心技术解密，可提供全栈的大数据+云原生平台咨询方案，请持续关注本套博客。如有任何学术交流，可随时联系。更多内容请关注《数据云技术社区》公众号。

1 管理控制中心机制(Informer联动)

1.1 包结构

1.2 controller的启动去脉

kube-controller-manager 作为集群的管理控制中心，维护集群中的所有控制器，对维持集群的稳定和自我修复，实现高可用，副本控制等起关键作用。
在controller-manager的Run函数部分调用了InformerFactory.Start的方法，Start方法初始化各种类型的informer，并且每个类型起了个informer.Run的goroutine。
三大核心函数：CreateControllerContext，StartControllers，ctx.InformerFactory.Start

k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go
// Run runs the KubeControllerManagerOptions.  This should never exit.
func Run(c *config.CompletedConfig) error {
  
        //1:拿到对kube-APIserver中资源的操作句柄,创建控制器上下文 
        ctx, err := CreateControllerContext(c, rootClientBuilder, clientBuilder, stop)

        //2:初始化的所有控制器（包括apiserver的客户端，informer的回调函数等等）
        if err := StartControllers(ctx, saTokenControllerInitFunc, NewControllerInitializers(ctx.LoopMode)); err != nil {
            glog.Fatalf("error starting controllers: %v", err)
        }
        
        //3:启动Informer,并完成Controller最终的启动以及资源监听机制
        ctx.InformerFactory.Start(ctx.Stop)
        close(ctx.InformersStarted)
}
复制代码

2 控制中心机制

2.1 控制中心入口之NewControllerManagerCommand

Kube-controller-manager的代码风格仍然是Cobra命令行框架。通过构造ControllerManagerCommand，然后执行command.Execute()函数。

k8s.io/kubernetes/cmd/kube-controller-manager/controller-manager.go
func main() {
    rand.Seed(time.Now().UTC().UnixNano())

    command := app.NewControllerManagerCommand()

    // TODO: once we switch everything over to Cobra commands, we can go back to calling
    // utilflag.InitFlags() (by removing its pflag.Parse() call). For now, we have to set the
    // normalize func and add the go flag set by hand.
    pflag.CommandLine.SetNormalizeFunc(utilflag.WordSepNormalizeFunc)
    pflag.CommandLine.AddGoFlagSet(goflag.CommandLine)
    // utilflag.InitFlags()
    logs.InitLogs()
    defer logs.FlushLogs()

    if err := command.Execute(); err != nil {
        fmt.Fprintf(os.Stderr, "%v\n", err)
        os.Exit(1)
    }
}
复制代码

2.2 控制中心之NewKubeControllerManagerOptions

构建ControllerManagerCommand基本的流程就是先构造NewControllerManagerCommand，添加Flags，执行Run函数。
NewKubeControllerManagerOptions是初始化controllerManager的参数，其中主要包括了各种controller的option

func NewControllerManagerCommand() *cobra.Command {

    s, err := options.NewKubeControllerManagerOptions()
    if err != nil {
        glog.Fatalf("unable to initialize command options: %v", err)
    }

    cmd := &cobra.Command{
        Use: "kube-controller-manager",
        Long: `The Kubernetes controller manager is a daemon that embeds
the core control loops shipped with Kubernetes. In applications of robotics and
automation, a control loop is a non-terminating loop that regulates the state of
the system. In Kubernetes, a controller is a control loop that watches the shared
state of the cluster through the apiserver and makes changes attempting to move the
current state towards the desired state. Examples of controllers that ship with
Kubernetes today are the replication controller, endpoints controller, namespace
controller, and serviceaccounts controller.`,
        Run: func(cmd *cobra.Command, args []string) {
            verflag.PrintAndExitIfRequested()
            utilflag.PrintFlags(cmd.Flags())

            c, err := s.Config(KnownControllers(), ControllersDisabledByDefault.List())
            if err != nil {
                fmt.Fprintf(os.Stderr, "%v\n", err)
                os.Exit(1)
            }

            if err := Run(c.Complete()); err != nil {
                fmt.Fprintf(os.Stderr, "%v\n", err)
                os.Exit(1)
            }
        },
    }
    s.AddFlags(cmd.Flags(), KnownControllers(), ControllersDisabledByDefault.List())

    return cmd
}

//初始化controllerManager的参数，其中主要包括了各种controller的option,例如DeploymentControllerOptions
//NewKubeControllerManagerOptions creates a new KubeControllerManagerOptions with a default config.
func NewKubeControllerManagerOptions() (*KubeControllerManagerOptions, error) {
    componentConfig, err := NewDefaultComponentConfig(ports.InsecureKubeControllerManagerPort)
    if err != nil {
        return nil, err
    }

    s := KubeControllerManagerOptions{
        Generic:         cmoptions.NewGenericControllerManagerConfigurationOptions(componentConfig.Generic),
        KubeCloudShared: cmoptions.NewKubeCloudSharedOptions(componentConfig.KubeCloudShared),
        AttachDetachController: &AttachDetachControllerOptions{
            ReconcilerSyncLoopPeriod: componentConfig.AttachDetachController.ReconcilerSyncLoopPeriod,
        },
        CSRSigningController: &CSRSigningControllerOptions{
            ClusterSigningCertFile: componentConfig.CSRSigningController.ClusterSigningCertFile,
            ClusterSigningKeyFile:  componentConfig.CSRSigningController.ClusterSigningKeyFile,
            ClusterSigningDuration: componentConfig.CSRSigningController.ClusterSigningDuration,
        },
        DaemonSetController: &DaemonSetControllerOptions{
            ConcurrentDaemonSetSyncs: componentConfig.DaemonSetController.ConcurrentDaemonSetSyncs,
        },
        DeploymentController: &DeploymentControllerOptions{
            ConcurrentDeploymentSyncs:      componentConfig.DeploymentController.ConcurrentDeploymentSyncs,
            DeploymentControllerSyncPeriod: componentConfig.DeploymentController.DeploymentControllerSyncPeriod,
        },
        DeprecatedFlags: &DeprecatedControllerOptions{
            RegisterRetryCount: componentConfig.DeprecatedController.RegisterRetryCount,
        },
        EndpointController: &EndpointControllerOptions{
            ConcurrentEndpointSyncs: componentConfig.EndpointController.ConcurrentEndpointSyncs,
        },
        GarbageCollectorController: &GarbageCollectorControllerOptions{
            ConcurrentGCSyncs:      componentConfig.GarbageCollectorController.ConcurrentGCSyncs,
            EnableGarbageCollector: componentConfig.GarbageCollectorController.EnableGarbageCollector,
        },
        HPAController: &HPAControllerOptions{
            HorizontalPodAutoscalerSyncPeriod:                   componentConfig.HPAController.HorizontalPodAutoscalerSyncPeriod,
            HorizontalPodAutoscalerUpscaleForbiddenWindow:       componentConfig.HPAController.HorizontalPodAutoscalerUpscaleForbiddenWindow,
            HorizontalPodAutoscalerDownscaleForbiddenWindow:     componentConfig.HPAController.HorizontalPodAutoscalerDownscaleForbiddenWindow,
            HorizontalPodAutoscalerDownscaleStabilizationWindow: componentConfig.HPAController.HorizontalPodAutoscalerDownscaleStabilizationWindow,
            HorizontalPodAutoscalerCPUInitializationPeriod:      componentConfig.HPAController.HorizontalPodAutoscalerCPUInitializationPeriod,
            HorizontalPodAutoscalerInitialReadinessDelay:        componentConfig.HPAController.HorizontalPodAutoscalerInitialReadinessDelay,
            HorizontalPodAutoscalerTolerance:                    componentConfig.HPAController.HorizontalPodAutoscalerTolerance,
            HorizontalPodAutoscalerUseRESTClients:               componentConfig.HPAController.HorizontalPodAutoscalerUseRESTClients,
        },
        JobController: &JobControllerOptions{
            ConcurrentJobSyncs: componentConfig.JobController.ConcurrentJobSyncs,
        },
        NamespaceController: &NamespaceControllerOptions{
            NamespaceSyncPeriod:      componentConfig.NamespaceController.NamespaceSyncPeriod,
            ConcurrentNamespaceSyncs: componentConfig.NamespaceController.ConcurrentNamespaceSyncs,
        },
        NodeIPAMController: &NodeIPAMControllerOptions{
            NodeCIDRMaskSize: componentConfig.NodeIPAMController.NodeCIDRMaskSize,
        },
        NodeLifecycleController: &NodeLifecycleControllerOptions{
            EnableTaintManager:     componentConfig.NodeLifecycleController.EnableTaintManager,
            NodeMonitorGracePeriod: componentConfig.NodeLifecycleController.NodeMonitorGracePeriod,
            NodeStartupGracePeriod: componentConfig.NodeLifecycleController.NodeStartupGracePeriod,
            PodEvictionTimeout:     componentConfig.NodeLifecycleController.PodEvictionTimeout,
        },
        PersistentVolumeBinderController: &PersistentVolumeBinderControllerOptions{
            PVClaimBinderSyncPeriod: componentConfig.PersistentVolumeBinderController.PVClaimBinderSyncPeriod,
            VolumeConfiguration:     componentConfig.PersistentVolumeBinderController.VolumeConfiguration,
        },
        PodGCController: &PodGCControllerOptions{
            TerminatedPodGCThreshold: componentConfig.PodGCController.TerminatedPodGCThreshold,
        },
        ReplicaSetController: &ReplicaSetControllerOptions{
            ConcurrentRSSyncs: componentConfig.ReplicaSetController.ConcurrentRSSyncs,
        },
        ReplicationController: &ReplicationControllerOptions{
            ConcurrentRCSyncs: componentConfig.ReplicationController.ConcurrentRCSyncs,
        },
        ResourceQuotaController: &ResourceQuotaControllerOptions{
            ResourceQuotaSyncPeriod:      componentConfig.ResourceQuotaController.ResourceQuotaSyncPeriod,
            ConcurrentResourceQuotaSyncs: componentConfig.ResourceQuotaController.ConcurrentResourceQuotaSyncs,
        },
        SAController: &SAControllerOptions{
            ConcurrentSATokenSyncs: componentConfig.SAController.ConcurrentSATokenSyncs,
        },
        ServiceController: &cmoptions.ServiceControllerOptions{
            ConcurrentServiceSyncs: componentConfig.ServiceController.ConcurrentServiceSyncs,
        },
        TTLAfterFinishedController: &TTLAfterFinishedControllerOptions{
            ConcurrentTTLSyncs: componentConfig.TTLAfterFinishedController.ConcurrentTTLSyncs,
        },
        SecureServing: apiserveroptions.NewSecureServingOptions().WithLoopback(),
        InsecureServing: (&apiserveroptions.DeprecatedInsecureServingOptions{
            BindAddress: net.ParseIP(componentConfig.Generic.Address),
            BindPort:    int(componentConfig.Generic.Port),
            BindNetwork: "tcp",
        }).WithLoopback(),
        Authentication: apiserveroptions.NewDelegatingAuthenticationOptions(),
        Authorization:  apiserveroptions.NewDelegatingAuthorizationOptions(),
    }

    s.Authentication.RemoteKubeConfigFileOptional = true
    s.Authorization.RemoteKubeConfigFileOptional = true
    s.Authorization.AlwaysAllowPaths = []string{"/healthz"}

    s.SecureServing.ServerCert.CertDirectory = "/var/run/kubernetes"
    s.SecureServing.ServerCert.PairName = "kube-controller-manager"
    s.SecureServing.BindPort = ports.KubeControllerManagerPort

    gcIgnoredResources := make([]kubectrlmgrconfig.GroupResource, 0, len(garbagecollector.DefaultIgnoredResources()))
    for r := range garbagecollector.DefaultIgnoredResources() {
        gcIgnoredResources = append(gcIgnoredResources, kubectrlmgrconfig.GroupResource{Group: r.Group, Resource: r.Resource})
    }

    s.GarbageCollectorController.GCIgnoredResources = gcIgnoredResources

    return &s, nil
}

复制代码

2.3 控制中心之command.Execute()

加载所有控制器，并将对应参数注入到控制器中

c, err := s.Config(KnownControllers(), ControllersDisabledByDefault.List())

//k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go
KnownControllers()中的NewControllerInitializers初始化所有的控制器

c, err := s.Config(KnownControllers(), ControllersDisabledByDefault.List())
func NewControllerInitializers(loopMode ControllerLoopMode) map[string]InitFunc {
    controllers := map[string]InitFunc{}
    controllers["endpoint"] = startEndpointController
    controllers["replicationcontroller"] = startReplicationController
    controllers["podgc"] = startPodGCController
    controllers["resourcequota"] = startResourceQuotaController
    controllers["namespace"] = startNamespaceController
    controllers["serviceaccount"] = startServiceAccountController
    controllers["garbagecollector"] = startGarbageCollectorController
    controllers["daemonset"] = startDaemonSetController
    controllers["job"] = startJobController
    controllers["deployment"] = startDeploymentController
    controllers["replicaset"] = startReplicaSetController
    controllers["horizontalpodautoscaling"] = startHPAController
    controllers["disruption"] = startDisruptionController
    controllers["statefulset"] = startStatefulSetController
    controllers["cronjob"] = startCronJobController
    controllers["csrsigning"] = startCSRSigningController
    controllers["csrapproving"] = startCSRApprovingController
    controllers["csrcleaner"] = startCSRCleanerController
    controllers["ttl"] = startTTLController
    controllers["bootstrapsigner"] = startBootstrapSignerController
    controllers["tokencleaner"] = startTokenCleanerController
    controllers["nodeipam"] = startNodeIpamController
    if loopMode == IncludeCloudLoops {
        controllers["service"] = startServiceController
        controllers["route"] = startRouteController
        // TODO: volume controller into the IncludeCloudLoops only set.
        // TODO: Separate cluster in cloud check from node lifecycle controller.
    }
    controllers["nodelifecycle"] = startNodeLifecycleController
    controllers["persistentvolume-binder"] = startPersistentVolumeBinderController
    controllers["attachdetach"] = startAttachDetachController
    controllers["persistentvolume-expander"] = startVolumeExpandController
    controllers["clusterrole-aggregation"] = startClusterRoleAggregrationController
    controllers["pvc-protection"] = startPVCProtectionController
    controllers["pv-protection"] = startPVProtectionController
    return controllers
}
复制代码

启动controller-manager的http服务和对应处理器，包括安全和非安全：BuildHandlerChain ,构造run的执行体。

k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go
// Run runs the KubeControllerManagerOptions.  This should never exit.
func Run(c *config.CompletedConfig) error {
  
        //1:拿到对kube-APIserver中资源的操作句柄,创建控制器上下文 
        ctx, err := CreateControllerContext(c, rootClientBuilder, clientBuilder, stop)

        //2:初始化的所有控制器（包括apiserver的客户端，informer的回调函数等等）
        if err := StartControllers(ctx, saTokenControllerInitFunc, NewControllerInitializers(ctx.LoopMode)); err != nil {
            glog.Fatalf("error starting controllers: %v", err)
        }
        
        //3:启动Informer,并完成Controller最终的启动以及资源监听机制
        ctx.InformerFactory.Start(ctx.Stop)
        close(ctx.InformersStarted)
}
复制代码

2.4 创建控制器上下文

拿到对kube-APIserver中资源的操作句柄
确认Kube-APIServer的健康（最多等待10s），然后拿获取连接
创建控制器上下文

func CreateControllerContext(s *config.CompletedConfig, rootClientBuilder, clientBuilder controller.ControllerClientBuilder, stop <-chan struct{}) (ControllerContext, error) {
    //拿到对APIServer资源的操作句柄
    versionedClient := rootClientBuilder.ClientOrDie("shared-informers")
    sharedInformers := informers.NewSharedInformerFactory(versionedClient, ResyncPeriod(s)())

    // If apiserver is not running we should wait for some time and fail only then. This is particularly
    // important when we start apiserver and controller manager at the same time.
    //gaogao note : 10s内检查APIserver服务是否可用
    if err := genericcontrollermanager.WaitForAPIServer(versionedClient, 10*time.Second); err != nil {
        return ControllerContext{}, fmt.Errorf("failed to wait for apiserver being healthy: %v", err)
    }

    // Use a discovery client capable of being refreshed.
    discoveryClient := rootClientBuilder.ClientOrDie("controller-discovery")
    //note:  DiscoveryClient = discoveryClient.Discovery()
    cachedClient := cacheddiscovery.NewMemCacheClient(discoveryClient.Discovery())
    restMapper := restmapper.NewDeferredDiscoveryRESTMapper(cachedClient)
    go wait.Until(func() {
        restMapper.Reset()
    }, 30*time.Second, stop)

    availableResources, err := GetAvailableResources(rootClientBuilder)
    if err != nil {
        return ControllerContext{}, err
    }

    cloud, loopMode, err := createCloudProvider(s.ComponentConfig.CloudProvider.Name, s.ComponentConfig.ExternalCloudVolumePlugin,
        s.ComponentConfig.CloudProvider.CloudConfigFile, s.ComponentConfig.KubeCloudShared.AllowUntaggedCloud, sharedInformers)
    if err != nil {
        return ControllerContext{}, err
    }

    ctx := ControllerContext{
        ClientBuilder:      clientBuilder,
        InformerFactory:    sharedInformers,
        ComponentConfig:    s.ComponentConfig,
        RESTMapper:         restMapper,
        AvailableResources: availableResources,
        Cloud:              cloud,
        LoopMode:           loopMode,
        Stop:               stop,
        InformersStarted:   make(chan struct{}),
        ResyncPeriod:       ResyncPeriod(s),
    }
    return ctx, nil
}
复制代码

2.5 初始化的所有控制器

构造controller manager option，并转化为Config对象，执行Run函数。
基于Config对象创建ControllerContext，其中包含InformerFactory。
基于ControllerContext运行各种controller，各种controller的定义在NewControllerInitializers中。
执行InformerFactory.Start(ControllerContext已经定义好controller)，实际执行了controllerContext.InformerFactory.Start(controllerContext.Stop)
每种controller都会构造自身的结构体并执行对应的Run函数。

func StartControllers(ctx ControllerContext, startSATokenController InitFunc, controllers map[string]InitFunc) error {
    ···
    for controllerName, initFn := range controllers {
        if !ctx.IsControllerEnabled(controllerName) {
            glog.Warningf("%q is disabled", controllerName)
            continue
        }

        time.Sleep(wait.Jitter(ctx.ComponentConfig.GenericComponent.ControllerStartInterval.Duration, ControllerStartJitter))

        glog.V(1).Infof("Starting %q", controllerName)
        
        //note : initFn为初始化controller是创建的初始化函数
        started, err := initFn(ctx)
        ···
    }

    return nil
}

复制代码

启动控制器和监听资源的事件

//SharedInformerFactory是一个informer工厂的接口定义。
// SharedInformerFactory a small interface to allow for adding an informer without an import cycle
type SharedInformerFactory interface {
    Start(stopCh <-chan struct{})
    InformerFor(obj runtime.Object, newFunc NewInformerFunc) cache.SharedIndexInformer
}

// Start initializes all requested informers.
func (f *sharedInformerFactory) Start(stopCh <-chan struct{}) {
    f.lock.Lock()
    defer f.lock.Unlock()

    for informerType, informer := range f.informers {
        if !f.startedInformers[informerType] {
            go informer.Run(stopCh)
            f.startedInformers[informerType] = true
        }
    }
}
复制代码

2.6 InformerFactory.Start的调用者controllerContext

InformerFactory实际上是SharedInformerFactory，具体的实现逻辑在client-go中的informer的实现机制。

controllerContext.InformerFactory.Start(controllerContext.Stop)
close(controllerContext.InformersStarted)
复制代码

2.7 总体架构图一览

管理控制中心机制源码调用关系

上图参考https://www.jianshu.com/p/ac9179007fe2，感谢原作者，本文主要融入新理解，全新解读。
复制代码

Informer与controller关系一览

3 再总结

Kube-controller-manager对应的cmd部分的调用流程如下：Main-->NewControllerManagerCommand--> Run(c.Complete(), wait.NeverStop)-->StartControllers-->initFn(ctx)-->startDeploymentController/startStatefulSetController-->InformerFactory.Start->sts.NewStatefulSetController.Run/dc.NewDeploymentController.Run-->pkg/controller。
其中CreateControllerContext函数用来创建各类型controller所需要使用的context，NewControllerInitializers初始化了各种类型的controller，其中就包括DeploymentController和StatefulSetController等。

4 最后

kube-controller-manager管理控制中心是和InformerFactory.Start遥相呼应的，因为Informer启动后,才能完成Controller最终的启动以及资源监听机制。实际上通过controllerContext.InformerFactory.Start(controllerContext.Stop) 也印证了。

转载于:https://juejin.im/post/5d614862f265da03b31bd897

weixin_30677617

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Kubernetes kube-controller-manager 控制中心机制源码深入剖析-Kubernetes商业环境实战

专注于大数据及容器云核心技术解密，可提供全栈的大数据+云原生平台咨询方案，请持续关注本套博客。如有任何学术交流，可随时联系。更多内容请关注《数据云技术社区》公众号。1 管理控制中心机制(Informer联动)1.1 包结构1.2 controller的启动去脉kube-controller-manager 作为集群的管理控制中心，维护集群中的所有控制器，对维持集群的稳定和自我修复，实...
复制链接

扫一扫