纯干货整理，一文看懂 kubebuilder 安装，部署以及源码controller-runtime等库分析

最新推荐文章于 2024-02-04 15:19:08 发布

Magiceses

最新推荐文章于 2024-02-04 15:19:08 发布

阅读量2.6k

点赞数

文章标签： go etcd kubernetes

本文链接：https://blog.csdn.net/weixin_43700106/article/details/115916231

版权

快速开始

构建go环境

curl -o /tmp/go1.13.14.linux-amd64.tar.gz https://dl.google.com/go/go1.13.14.linux-amd64.tar.gz
rm -rf /usr/local/go/ && tar -C /usr/local/ -zxf /tmp/go1.13.14.linux-amd64.tar.gz
cat <<"EOF" | tee -a ~/.bashrc

# add by chengql start
export PATH=$PATH:/usr/local/go/bin
export PATH=$PATH:$(go env GOPATH)/bin
#export GO111MODULE=on
export GOPROXY=https://goproxy.io,direct
# add by chengql end
EOF

export PATH=$PATH:/usr/local/go/bin
export PATH=$PATH:$(go env GOPATH)/bin
export GO111MODULE=on
export GOPROXY=https://goproxy.io,direct

go get -u github.com/posener/complete/gocomplete
gocomplete -y -install
go get -u github.com/go-delve/delve/cmd/dlv

安装kubebuilder

os=$(go env GOOS)
arch=$(go env GOARCH)

# 下载 kubebuilder 并解压到 tmp 目录中
curl -L https://go.kubebuilder.io/dl/2.3.1/${os}/${arch} | tar -xz -C /tmp/

# 将 kubebuilder 移动到一个长期的路径，并将其加入环境变量 path 中 
# （如果你把 kubebuilder 放在别的地方，你需要额外设置 KUBEBUILDER_ASSETS 环境变量）
sudo mv /tmp/kubebuilder_2.3.1_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin

初始化kubebuilder

mkdir $GOPATH/src/example
cd $GOPATH/src/example

kubebuilder init --domain my.domain

注意：

1. 如果你不在GOPATH目录，你需要执行 go mod init <modulename> 去告诉kubebuilder和go你需要的记录目录

2. 如果你遇到了 cannot find package ... (from $GOROOT) 的错误，可以执行export GO111MODULE=on 来解决

➜  kubebuilder-example go mod init kubebuilder-example
go: creating new go.mod: module kubebuilder-example
➜  kubebuilder-example ls
go.mod

创建API

kubebuilder create api --group webapp --version v1 --kind Guestbook

sample:
apiVersion: webapp.my.domain/v1
kind: Guestbook
metadata:
  name: guestbook-sample
spec:
  # Add fields here
  foo: bar

➜  kubebuilder-example ll
total 128
-rw-------   1 stark  staff   795B  3 15 15:30 Dockerfile
-rw-------   1 stark  staff   2.0K  3 15 15:30 Makefile
-rw-------   1 stark  staff   116B  3 15 15:31 PROJECT
drwx------   3 stark  staff    96B  3 15 15:31 api
drwxr-xr-x   3 stark  staff    96B  3 15 15:31 bin
drwx------  10 stark  staff   320B  3 15 15:31 config
drwx------   4 stark  staff   128B  3 15 15:31 controllers
-rw-r--r--   1 stark  staff   238B  3 15 15:31 go.mod
-rw-r--r--   1 stark  staff    43K  3 15 15:30 go.sum
drwx------   3 stark  staff    96B  3 15 15:30 hack
-rw-------   1 stark  staff   2.3K  3 15 15:31 main.go

➜  kubebuilder-example tree
.
├── Dockerfile # 制作crd-controller镜像的ockerfile
├── Makefile # make编译文件
├── PROJECT # 项目元数据
├── api
│   └── v1
│       ├── groupversion_info.go # GVK信息、scheme生成的方法都在这里
│       ├── guestbook_types.go # 自定义CRD结构
│       └── zz_generated.deepcopy.go # 资源对象的操作一开始都是建立在deepcopy出来的复制对象身上的
├── bin
│   └── manager # go打包的二进制文件
├── config # 所有最终生成的需要kubectl apply的的资源，按照功能进行分片成不同的目录，这里有些地方可以做些自定义的配置
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── crd # crd的配置
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_guestbooks.yaml
│   │       └── webhook_in_guestbooks.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   ├── manager # manager的deployment在这里
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus # metric暴露
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac # rbac授权
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── guestbook_editor_role.yaml
│   │   ├── guestbook_viewer_role.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   └── role_binding.yaml
│   ├── samples # webapp resource sample
│   │   └── webapp_v1_guestbook.yaml
│   └── webhook # webapp webhook Service，用来接收APIServer转发而来的webhook请求
│       ├── kustomization.yaml
│       ├── kustomizeconfig.yaml
│       └── service.yaml
├── controllers
│   ├── guestbook_controller.go # CRD controller的核心逻辑在这里(Reconcile)
│   └── suite_test.go 
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
└── main.go # Entrypoint

15 directories, 42 files

安装crd

make install

make install 的时候你可能遇到下面的问题：

/bin/sh: kustomize: command not found
error: no objects passed to apply
make: *** [install] Error 1

执行 go get github.com/kubernetes-sigs/kustomize 即可，如果不能下载，go env -w GOPROXY=https://goproxy.io,direct ，换下代理重新试下

如果还是无法解决：export PATH= $P A T H :$ (go env GOPATH)/bin 试下

kubectl apply -f config/samples/ 部署cr，可以看到有日志输出

build 镜像

make docker-build docker-push IMG=<some-registry>/<project-name>:tag

安装 docker 的步骤
1. wget https://download.docker.com/linux/static/stable/x86_64/docker-18.03.1-ce.tgz
2. tar xzvf docker-18.03.1-ce.tgz
3. cp docker/* /usr/bin/
4. vi /usr/lib/systemd/system/docker.service 填写以下内容

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

5. systemctl daemon-reload
 systemctl start docker.service

部署运行

make deploy IMG=<some-registry>/<project-name>:tag

make run

源码分析

入口：main.go

package main

import (
	"flag"
	"os"

	"k8s.io/apimachinery/pkg/runtime"
	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
	_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"

	webappv1 "kubebuilder-example/api/v1"
	"kubebuilder-example/controllers"
	// +kubebuilder:scaffold:imports
)

var (
	scheme   = runtime.NewScheme()
	setupLog = ctrl.Log.WithName("setup")
)

func init() {
  // go init 顺序：https://www.jianshu.com/p/5f7338fcdec1 
  // 注册 crd scheme
	_ = clientgoscheme.AddToScheme(scheme)

	_ = webappv1.AddToScheme(scheme)
	// +kubebuilder:scaffold:scheme
}

func main() {
	var metricsAddr string
	var enableLeaderElection bool
  // 命令行解析包，有兴趣可见：https://www.cnblogs.com/aaronthon/p/10883711.html
	flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "The address the metric endpoint binds to.")
	flag.BoolVar(&enableLeaderElection, "enable-leader-election", false,
		"Enable leader election for controller manager. "+
			"Enabling this will ensure there is only one active controller manager.")
	flag.Parse()

	ctrl.SetLogger(zap.New(zap.UseDevMode(true)))
  
  // 生成一个manager，初始化所需的各种配置，实现了Manager接口
	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:             scheme,
		MetricsBindAddress: metricsAddr,
		Port:               9443,
		LeaderElection:     enableLeaderElection,
		LeaderElectionID:   "5a48bfe8.my.domain",
	})
	if err != nil {
		setupLog.Error(err, "unable to start manager")
		os.Exit(1)
	}

  // 这里看似没有涉及到controller，实际上在SetupWithManager中，使用了生成器模式，最终实现了
  // manager -> controller -> reconciler 的对象层级结构
  // 主体分为两部分，一是配置manager，二是启动manager
	if err = (&controllers.GuestbookReconciler{
		Client: mgr.GetClient(),
		Log:    ctrl.Log.WithName("controllers").WithName("Guestbook"),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "Guestbook")
		os.Exit(1)
	}
	// +kubebuilder:scaffold:builder

	setupLog.Info("starting manager")
  // 启动manager，主要是启动manager中注册的controller
	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
		setupLog.Error(err, "problem running manager")
		os.Exit(1)
	}
}

这里面主要注意三个步骤：

New Manager – 配置Manager
Guestbook Reconciler – Reconcile
Manager Start – 启动Manager

分别分析一下：

配置Manager

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:             scheme,
		MetricsBindAddress: metricsAddr,
		Port:               9443,
		LeaderElection:     enableLeaderElection,
		LeaderElectionID:   "5a48bfe8.my.domain",
	})

// New returns a new Manager for creating Controllers.
func New(config *rest.Config, options Options) (Manager, error) {
	// Initialize a rest.config if none was specified
	if config == nil {
		return nil, fmt.Errorf("must specify Config")
	}

	// Set default values for options fields
	options = setOptionsDefaults(options)

	// Create the mapper provider
	mapper, err := options.MapperProvider(config)
	if err != nil {
		log.Error(err, "Failed to get API Group-Resources")
		return nil, err
	}

	// 创建Cache对象，用做client的读请求，以及生成informer
	cache, err := options.NewCache(config, cache.Options{Scheme: options.Scheme, Mapper: mapper, Resync: options.SyncPeriod, Namespace: options.Namespace})
	if err != nil {
		return nil, err
	}
  // 创建读请求的client，即apiReader，读请求走的Cache
	apiReader, err := client.New(config, client.Options{Scheme: options.Scheme, Mapper: mapper})
	if err != nil {
		return nil, err
	}
  // 创建写请求的client，写请求直连APIServer
	writeObj, err := options.NewClient(cache, config, client.Options{Scheme: options.Scheme, Mapper: mapper})
	if err != nil {
		return nil, err
	}
  
	// recorderProvider,记录event事件用的，kubectl describe可用到
	recorderProvider, err := options.newRecorderProvider(config, options.Scheme, log.WithName("events"), options.EventBroadcaster)
	if err != nil {
		return nil, err
	}

  // controller多副本leader选举使用的
	resourceLock, err := options.newResourceLock(config, recorderProvider, leaderelection.Options{
		LeaderElection:          options.LeaderElection,
		LeaderElectionID:        options.LeaderElectionID,
		LeaderElectionNamespace: options.LeaderElectionNamespace,
	})
	if err != nil {
		return nil, err
	}

	// 暴露/metrics给prometheus使用
	metricsListener, err := options.newMetricsListener(options.MetricsBindAddress)
	if err != nil {
		return nil, err
	}

	// Create health probes listener. This will throw an error if the bind
	// address is invalid or already in use.
	healthProbeListener, err := options.newHealthProbeListener(options.HealthProbeBindAddress)
	if err != nil {
		return nil, err
	}

	stop := make(chan struct{})

	return &controllerManager{
		config:                config,
		scheme:                options.Scheme,
		cache:                 cache,
		fieldIndexes:          cache,
		client:                writeObj,
		apiReader:             apiReader,
		recorderProvider:      recorderProvider,
		resourceLock:          resourceLock,
		mapper:                mapper,
		metricsListener:       metricsListener,
		internalStop:          stop,
		internalStopper:       stop,
		port:                  options.Port,
		host:                  options.Host,
		certDir:               options.CertDir,
		leaseDuration:         *options.LeaseDuration,
		renewDeadline:         *options.RenewDeadline,
		retryPeriod:           *options.RetryPeriod,
		healthProbeListener:   healthProbeListener,
		readinessEndpointName: options.ReadinessEndpointName,
		livenessEndpointName:  options.LivenessEndpointName,
	}, nil
}

我们来看下NewCache

// NewCache
// 这里就是根据配置，来生成所需要监测的每种GVK对应的Informer
// New initializes and returns a new Cache.
func New(config *rest.Config, opts Options) (Cache, error) {
	opts, err := defaultOpts(config, opts)
	if err != nil {
		return nil, err
	}
	im := internal.NewInformersMap(config, opts.Scheme, opts.Mapper, *opts.Resync, opts.Namespace)
	return &informerCache{InformersMap: im}, nil
}

func NewInformersMap(config *rest.Config,
	scheme *runtime.Scheme,
	mapper meta.RESTMapper,
	resync time.Duration,
	namespace string) *InformersMap {

	return &InformersMap{
		structured:   newStructuredInformersMap(config, scheme, mapper, resync, namespace),
		unstructured: newUnstructuredInformersMap(config, scheme, mapper, resync, namespace),

		Scheme: scheme,
	}
}

// newStructuredInformersMap creates a new InformersMap for structured objects.
func newStructuredInformersMap(config *rest.Config, scheme *runtime.Scheme, mapper meta.RESTMapper, resync time.Duration, namespace string) *specificInformersMap {
	return newSpecificInformersMap(config, scheme, mapper, resync, namespace, createStructuredListWatch)
}

// newUnstructuredInformersMap creates a new InformersMap for unstructured objects.
func newUnstructuredInformersMap(config *rest.Config, scheme *runtime.Scheme, mapper meta.RESTMapper, resync time.Duration, namespace string) *specificInformersMap {
	return newSpecificInformersMap(config, scheme, mapper, resync, namespace, createUnstructuredListWatch)
}

// newSpecificInformersMap returns a new specificInformersMap (like
// the generical InformersMap, except that it doesn't implement WaitForCacheSync).
func newSpecificInformersMap(config *rest.Config,
	scheme *runtime.Scheme,
	mapper meta.RESTMapper,
	resync time.Duration,
	namespace string,
	createListWatcher createListWatcherFunc) *specificInformersMap {
	ip := &specificInformersMap{
		config:            config,
		Scheme:            scheme,
		mapper:            mapper,
		informersByGVK:    make(map[schema.GroupVersionKind]*MapEntry),
		codecs:            serializer.NewCodecFactory(scheme),
		paramCodec:        runtime.NewParameterCodec(scheme),
		resync:            resync,
		startWait:         make(chan struct{}),
		createListWatcher: createListWatcher,
		namespace:         namespace,
	}
	return ip
}

createStructuredListWatch

// newListWatch returns a new ListWatch object that can be used to create a SharedIndexInformer.
func createStructuredListWatch(gvk schema.GroupVersionKind, ip *specificInformersMap) (*cache.ListWatch, error) {
	// Kubernetes APIs work against Resources, not GroupVersionKinds.  Map the
	// groupVersionKind to the Resource API we will use.
	mapping, err := ip.mapper.RESTMapping(gvk.GroupKind(), gvk.Version)
	if err != nil {
		return nil, err
	}

	client, err := apiutil.RESTClientForGVK(gvk, ip.config, ip.codecs)
	if err != nil {
		return nil, err
	}
	listGVK := gvk.GroupVersion().WithKind(gvk.Kind + "List")
	listObj, err := ip.Scheme.New(listGVK)
	if err != nil {
		return nil, err
	}

	// Create a new ListWatch for the obj
	return &cache.ListWatch{
		ListFunc: func(opts metav1.ListOptions) (runtime.Object, error) {
			res := listObj.DeepCopyObject()
			isNamespaceScoped := ip.namespace != "" && mapping.Scope.Name() != meta.RESTScopeNameRoot
			err := client.Get().NamespaceIfScoped(ip.namespace, isNamespaceScoped).Resource(mapping.Resource.Resource).VersionedParams(&opts, ip.paramCodec).Do().Into(res)
			return res, err
		},
		// Setup the watch function
		WatchFunc: func(opts metav1.ListOptions) (watch.Interface, error) {
			// Watch needs to be set to true separately
			opts.Watch = true
			isNamespaceScoped := ip.namespace != "" && mapping.Scope.Name() != meta.RESTScopeNameRoot
			return client.Get().NamespaceIfScoped(ip.namespace, isNamespaceScoped).Resource(mapping.Resource.Resource).VersionedParams(&opts, ip.paramCodec).Watch()
		},
	}, nil
}

createUnstructuredListWatch

func createUnstructuredListWatch(gvk schema.GroupVersionKind, ip *specificInformersMap) (*cache.ListWatch, error) {
	// Kubernetes APIs work against Resources, not GroupVersionKinds.  Map the
	// groupVersionKind to the Resource API we will use.
	mapping, err := ip.mapper.RESTMapping(gvk.GroupKind(), gvk.Version)
	if err != nil {
		return nil, err
	}
	dynamicClient, err := dynamic.NewForConfig(ip.config)
	if err != nil {
		return nil, err
	}

	// Create a new ListWatch for the obj
	return &cache.ListWatch{
		ListFunc: func(opts metav1.ListOptions) (runtime.Object, error) {
			if ip.namespace != "" && mapping.Scope.Name() != meta.RESTScopeNameRoot {
				return dynamicClient.Resource(mapping.Resource).Namespace(ip.namespace).List(opts)
			}
			return dynamicClient.Resource(mapping.Resource).List(opts)
		},
		// Setup the watch function
		WatchFunc: func(opts metav1.ListOptions) (watch.Interface, error) {
			// Watch needs to be set to true separately
			opts.Watch = true
			if ip.namespace != "" && mapping.Scope.Name() != meta.RESTScopeNameRoot {
				return dynamicClient.Resource(mapping.Resource).Namespace(ip.namespace).Watch(opts)
			}
			return dynamicClient.Resource(mapping.Resource).Watch(opts)
		},
	}, nil
}

NewClient

// NewClient
// defaultNewClient creates the default caching client
func defaultNewClient(cache cache.Cache, config *rest.Config, options client.Options) (client.Client, error) {
	// Create the Client for Write operations.
	// 写操作的直连APIServer的client
	c, err := client.New(config, options) // 见下面New内容
	if err != nil {
		return nil, err
	}

	return &client.DelegatingClient{
		Reader: &client.DelegatingReader{
		  // 读操作走manager里面的cache
			CacheReader:  cache,
			// 写操作的直连client
			ClientReader: c,
		},
		Writer:       c,
		StatusClient: c,
	}, nil
}

// 利用scheme来取得给定的资源type所属的GVK。
// 简而言之，这里的client是通过scheme与APIServer直接进行序列化和反序列化交互的
func New(config *rest.Config, options Options) (Client, error) {
	if config == nil {
		return nil, fmt.Errorf("must provide non-nil rest.Config to client.New")
	}

	// Init a scheme if none provided
	if options.Scheme == nil {
		options.Scheme = scheme.Scheme
	}

	// Init a Mapper if none provided
	if options.Mapper == nil {
		var err error
		options.Mapper, err = apiutil.NewDynamicRESTMapper(config)
		if err != nil {
			return nil, err
		}
	}

	dynamicClient, err := dynamic.NewForConfig(config)
	if err != nil {
		return nil, err
	}

	c := &client{
		typedClient: typedClient{
			cache: clientCache{
				config:         config,
				scheme:         options.Scheme,
				mapper:         options.Mapper,
				codecs:         serializer.NewCodecFactory(options.Scheme),
				resourceByType: make(map[reflect.Type]*resourceMeta),
			},
			paramCodec: runtime.NewParameterCodec(options.Scheme),
		},
		unstructuredClient: unstructuredClient{
			client:     dynamicClient,
			restMapper: options.Mapper,
		},
	}

	return c, nil
}

Reconciler

reconciler即controller，命名为reconciler，意为协调器更贴切，控制器的核心逻辑在这里面

	if err = (&controllers.GuestbookReconciler{
		Client: mgr.GetClient(),
		Log:    ctrl.Log.WithName("controllers").WithName("Guestbook"),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "Guestbook")
		os.Exit(1)
	}

func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.Guestbook{}).
		Complete(r)
}

// Complete方法是用作生成Builder,Builder最主要的是doController()和doWatch()方法，分别来看下
func (blder *Builder) Build(r reconcile.Reconciler) (controller.Controller, error) {
	if r == nil {
		return nil, fmt.Errorf("must provide a non-nil Reconciler")
	}
	if blder.mgr == nil {
		return nil, fmt.Errorf("must provide a non-nil Manager")
	}

	// Set the Config
	blder.loadRestConfig()

	// Set the ControllerManagedBy
  //  doController中新建了一个controller，reconciler作为入参，将controller和reconciler联系起来
	if err := blder.doController(r); err != nil {
		return nil, err
	}

	// Set the Watch
	if err := blder.doWatch(); err != nil {
		return nil, err
	}

	return blder.ctrl, nil

doController()

可以看出，doController方法是生成Controller，并将其注册进Manager的外层主体进行托管

func (blder *Builder) doController(r reconcile.Reconciler) error {
	name, err := blder.getControllerName()
	if err != nil {
		return err
	}
	ctrlOptions := blder.ctrlOptions
	ctrlOptions.Reconciler = r
	blder.ctrl, err = newController(name, blder.mgr, ctrlOptions)
	return err
}

var newController = controller.New

func New(name string, mgr manager.Manager, options Options) (Controller, error) {
	if options.Reconciler == nil {
		return nil, fmt.Errorf("must specify Reconciler")
	}

	if len(name) == 0 {
		return nil, fmt.Errorf("must specify Name for Controller")
	}

	if options.MaxConcurrentReconciles <= 0 {
		options.MaxConcurrentReconciles = 1
	}

	// Inject dependencies into Reconciler
	if err := mgr.SetFields(options.Reconciler); err != nil {
		return nil, err
	}

	// Create controller with dependencies set
	c := &controller.Controller{
    // 将reconciler赋值给Do，在下面的分析中可以看到对controller.Do.Reconile的调用
		Do:       options.Reconciler,
		Cache:    mgr.GetCache(),
		Config:   mgr.GetConfig(),
		Scheme:   mgr.GetScheme(),
		Client:   mgr.GetClient(),
		Recorder: mgr.GetEventRecorderFor(name),
    // 赋值了新建queue的方法，带有限速功能
		MakeQueue: func() workqueue.RateLimitingInterface {
			return workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), name)
		},
		MaxConcurrentReconciles: options.MaxConcurrentReconciles,
		Name:                    name,
	}

	// Add the controller as a Manager components
  // Add方法完成controller和manager关联
	return c, mgr.Add(c)
}

// Add sets dependencies on i, and adds it to the list of Runnables to start.
func (cm *controllerManager) Add(r Runnable) error {
	cm.mu.Lock()
	defer cm.mu.Unlock()

	// Set dependencies on the object
	if err := cm.SetFields(r); err != nil {
		return err
	}

	var shouldStart bool

	// Add the runnable to the leader election or the non-leaderelection list
	if leRunnable, ok := r.(LeaderElectionRunnable); ok && !leRunnable.NeedLeaderElection() {
		shouldStart = cm.started
    // 将r也就是上文提到的controller添加到nonLeaderElectionRunnables中，这里
    // 就完成了manager和controller的关联
		cm.nonLeaderElectionRunnables = append(cm.nonLeaderElectionRunnables, r)
	} else {
		shouldStart = cm.startedLeader
    // 将r也就是上文提到的controller添加到LeaderElectionRunnables中，这里
    // 就完成了manager和controller的关联
		cm.leaderElectionRunnables = append(cm.leaderElectionRunnables, r)
	}

  // 如果已经启动了，那么直接start即可将上述controller纳入管理
	if shouldStart {
		// If already started, start the controller
		go func() {
			if err := r.Start(cm.internalStop); err != nil {
				cm.errSignal.SignalError(err)
			}
		}()
	}

	return nil
}
/*
到这里完成了manager -> controller -> reconciler的对象关联，下面将看下manager如何启动controller，controller会启动资源的事件处理。
*/

其中，Controller结构体实例内包含的字段如下

c := &controller.Controller{
    // Reconciler只有一个接口方法Reconcile()，这个方法是CRD控制的核心逻辑，kubebuilder已经自动生成，但里面的逻辑需要填充，见下面
    Do:       options.Reconciler,  
    // Cache用来对接informer检测GVR状态并保存在缓存中，提供用作读
		Cache:    mgr.GetCache(),
		Config:   mgr.GetConfig(),
    // 用作资源实例Type的正反序列化
		Scheme:   mgr.GetScheme(),
    // Client用作写请求，直连APIServer
		Client:   mgr.GetClient(),
    // 记录event
		Recorder: mgr.GetEventRecorderFor(name),
    // workqueue，限速
		MakeQueue: func() workqueue.RateLimitingInterface {
			return workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), name)
		},
    // 协调器的并发数限制
		MaxConcurrentReconciles: options.MaxConcurrentReconciles,
		Name:                    name,
	}

最终会执行到，这个先跳过，后面会有分析

func (r *GuestbookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
	_ = context.Background()
	_ = r.Log.WithValues("guestbook", req.NamespacedName)

	// your logic here

	return ctrl.Result{}, nil
}

这里面逻辑需要自己实现

doWatch()

func (blder *Builder) doWatch() error {
	// Reconcile type
	src := &source.Kind{Type: blder.apiType}
	hdler := &handler.EnqueueRequestForObject{}
  // Watch CRD资源的变更请求
	err := blder.ctrl.Watch(src, hdler, blder.predicates...)
	if err != nil {
		return err
	}

	// Watch 被CRD管理的own resource的变更请求
	for _, obj := range blder.managedObjects {
		src := &source.Kind{Type: obj}
		hdler := &handler.EnqueueRequestForOwner{
			OwnerType:    blder.apiType,
			IsController: true,
		}
		if err := blder.ctrl.Watch(src, hdler, blder.predicates...); err != nil {
			return err
		}
	}

	// Do the watch requests
	for _, w := range blder.watchRequest {
		if err := blder.ctrl.Watch(w.src, w.eventhandler, blder.predicates...); err != nil {
			return err
		}

	}
	return nil
}

doWatch()主要干两件事：watch CRD 资源的变更，以及watch CRD 资源的own resouces的变更.watch到变更之后下一步做什么呢？当然是交给handler来处理，来看一下这里第四行代码中生成的handler是做什么的。

/*
Copyright 2018 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package handler

import (
	"k8s.io/apimachinery/pkg/types"
	"k8s.io/client-go/util/workqueue"
	"sigs.k8s.io/controller-runtime/pkg/event"
	logf "sigs.k8s.io/controller-runtime/pkg/internal/log"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

var enqueueLog = logf.RuntimeLog.WithName("eventhandler").WithName("EnqueueRequestForObject")

var _ EventHandler = &EnqueueRequestForObject{}

// EnqueueRequestForObject enqueues a Request containing the Name and Namespace of the object that is the source of the Event.
// (e.g. the created / deleted / updated objects Name and Namespace).  handler.EnqueueRequestForObject is used by almost all
// Controllers that have associated Resources (e.g. CRDs) to reconcile the associated Resource.
type EnqueueRequestForObject struct{}

// Create implements EventHandler
func (e *EnqueueRequestForObject) Create(evt event.CreateEvent, q workqueue.RateLimitingInterface) {
	if evt.Meta == nil {
		enqueueLog.Error(nil, "CreateEvent received with no metadata", "event", evt)
		return
	}
	q.Add(reconcile.Request{NamespacedName: types.NamespacedName{
		Name:      evt.Meta.GetName(),
		Namespace: evt.Meta.GetNamespace(),
	}})
}

// Update implements EventHandler
func (e *EnqueueRequestForObject) Update(evt event.UpdateEvent, q workqueue.RateLimitingInterface) {
	if evt.MetaOld != nil {
		q.Add(reconcile.Request{NamespacedName: types.NamespacedName{
			Name:      evt.MetaOld.GetName(),
			Namespace: evt.MetaOld.GetNamespace(),
		}})
	} else {
		enqueueLog.Error(nil, "UpdateEvent received with no old metadata", "event", evt)
	}

	if evt.MetaNew != nil {
		q.Add(reconcile.Request{NamespacedName: types.NamespacedName{
			Name:      evt.MetaNew.GetName(),
			Namespace: evt.MetaNew.GetNamespace(),
		}})
	} else {
		enqueueLog.Error(nil, "UpdateEvent received with no new metadata", "event", evt)
	}
}

// Delete implements EventHandler
func (e *EnqueueRequestForObject) Delete(evt event.DeleteEvent, q workqueue.RateLimitingInterface) {
	if evt.Meta == nil {
		enqueueLog.Error(nil, "DeleteEvent received with no metadata", "event", evt)
		return
	}
	q.Add(reconcile.Request{NamespacedName: types.NamespacedName{
		Name:      evt.Meta.GetName(),
		Namespace: evt.Meta.GetNamespace(),
	}})
}

// Generic implements EventHandler
func (e *EnqueueRequestForObject) Generic(evt event.GenericEvent, q workqueue.RateLimitingInterface) {
	if evt.Meta == nil {
		enqueueLog.Error(nil, "GenericEvent received with no metadata", "event", evt)
		return
	}
	q.Add(reconcile.Request{NamespacedName: types.NamespacedName{
		Name:      evt.Meta.GetName(),
		Namespace: evt.Meta.GetNamespace(),
	}})
}

不出意外，handler会增删查的写请求的对象的NamespacedName，压入workqueue里面，与此同时，在另一头检测workqueue的协调器Reconciler默默地开始运转。

需要准备的注册工作都做完了，下面就要回到main.go中，启动外层托管组件Manager了。

启动Manager

上面ctrl.NewManager()方法最终返回的是controllerManager{}对象的指针，来controllerManager里面找一下Start()方法.

func (cm *controllerManager) Start(stop <-chan struct{}) error {
	// join the passed-in stop channel as an upstream feeding into cm.internalStopper
	defer close(cm.internalStopper)

	// initialize this here so that we reset the signal channel state on every start
	cm.errSignal = &errSignaler{errSignal: make(chan struct{})}

	// Metrics should be served whether the controller is leader or not.
	// (If we don't serve metrics for non-leaders, prometheus will still scrape
	// the pod but will get a connection refused)
	if cm.metricsListener != nil {
		go cm.serveMetrics(cm.internalStop)
	}

	// Serve health probes
	if cm.healthProbeListener != nil {
		go cm.serveHealthProbes(cm.internalStop)
	}

	go cm.startNonLeaderElectionRunnables()

	if cm.resourceLock != nil {
		err := cm.startLeaderElection()
		if err != nil {
			return err
		}
	} else {
		go cm.startLeaderElectionRunnables()
	}

	select {
	case <-stop:
		// We are done
		return nil
	case <-cm.errSignal.GotError():
		// Error starting a controller
		return cm.errSignal.Error()
	}
}

// 核心是启动Informer, waitForCache ==> cm.startCache = cm.cache.Start ==> InformersMap.Start ==> 
// InformersMap.structured/unstructured.Start ==> Informer.Run
func (cm *controllerManager) startLeaderElectionRunnables() {
	cm.mu.Lock()
	defer cm.mu.Unlock()

	cm.waitForCache()

	// Start the leader election Runnables after the cache has synced
	for _, c := range cm.leaderElectionRunnables {
		// Controllers block, but we want to return an error if any have an error starting.
		// Write any Start errors to a channel so we can return them
		ctrl := c
		go func() {
			if err := ctrl.Start(cm.internalStop); err != nil {
				cm.errSignal.SignalError(err)
			}
			// we use %T here because we don't have a good stand-in for "name",
			// and the full runnable might not serialize (mutexes, etc)
			log.V(1).Info("leader-election runnable finished", "runnable type", fmt.Sprintf("%T", ctrl))
		}()
	}

	cm.startedLeader = true
}

waitForCache的主要作用是启动Cache和等待Cache的首次同步完成。

启动cache的步骤则包括：创建FIFO queue、初始化informer、reflector、LocalStorage cache、index索引等，详细步骤的分析，需要查看client-go的代码。

sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go

func (c *Controller) Start(stop <-chan struct{}) error {
	// use an IIFE to get proper lock handling
	// but lock outside to get proper handling of the queue shutdown
	c.mu.Lock()

	c.Queue = c.MakeQueue()
	defer c.Queue.ShutDown() // needs to be outside the iife so that we shutdown after the stop channel is closed

	err := func() error {
		defer c.mu.Unlock()

		// TODO(pwittrock): Reconsider HandleCrash
		defer utilruntime.HandleCrash()

		// NB(directxman12): launch the sources *before* trying to wait for the
		// caches to sync so that they have a chance to register their intendeded
		// caches.
		for _, watch := range c.watches {
			log.Info("Starting EventSource", "controller", c.Name, "source", watch.src)
			if err := watch.src.Start(watch.handler, c.Queue, watch.predicates...); err != nil {
				return err
			}
		}

		// Start the SharedIndexInformer factories to begin populating the SharedIndexInformer caches
		log.Info("Starting Controller", "controller", c.Name)

		// Wait for the caches to be synced before starting workers
		if c.WaitForCacheSync == nil {
			c.WaitForCacheSync = c.Cache.WaitForCacheSync
		}
		if ok := c.WaitForCacheSync(stop); !ok {
			// This code is unreachable right now since WaitForCacheSync will never return an error
			// Leaving it here because that could happen in the future
			err := fmt.Errorf("failed to wait for %s caches to sync", c.Name)
			log.Error(err, "Could not wait for Cache to sync", "controller", c.Name)
			return err
		}

    // 很重要的一个抖动参数，可以认为是一个冷却时间，目前还没有用到
		if c.JitterPeriod == 0 {
			c.JitterPeriod = 1 * time.Second
		}

		// Launch workers to process resources
		log.Info("Starting workers", "controller", c.Name, "worker count", c.MaxConcurrentReconciles)
		for i := 0; i < c.MaxConcurrentReconciles; i++ {
			// Process work items
      // 到这里便是启动了MaxConcurrentReconciles个worker
			go wait.Until(c.worker, c.JitterPeriod, stop)
		}

		c.Started = true
		return nil
	}()
	if err != nil {
		return err
	}

	<-stop
	log.Info("Stopping workers", "controller", c.Name)
	return nil
}
/*
controller-runtime整体上可以认为启动结束，收尾函数是wait.Until。这个函数的功能是1） 当接收到stop(channel变量)信号时，函数退出，否则2）运行c.worker函数，如果c.woker退出，间隔JitterPeriod后，再次运行c.worker。这里的JitterPeriod是一个比较重要的参数，它直接影响了问题中提及到的日志刷新频率。
*/

// worker runs a worker thread that just dequeues items, processes them, and marks them done.
// It enforces that the reconcileHandler is never invoked concurrently with the same object.
func (c *Controller) worker() {
  // 引入一个for循环，当processNextWorkItem返回true时，重复执行，如果返回了 false，则退出worker函数。
	for c.processNextWorkItem() {
	}
}

/*
结合上文中的wait.Until函数，达到了效果1)如果processNextWorkItem返回true时，Until函数暂不生效，逻辑一直在for循环的判断中。2) 如果processNextWorkItem返回false时，跳出了for循环，worker函数也会返回，这时Until函数生效，在JitterPeriod时间间隔后重新调用worker函数，即在1s后重新调度函数。如前文所述，这里会影响日志的刷新频率，前提是processNextWorkItem返回了false。继续看下processNextWorkItem中的逻辑是怎样的。
*/

// processNextWorkItem will read a single work item off the workqueue and
// attempt to process it, by calling the reconcileHandler.
func (c *Controller) processNextWorkItem() bool {
  // 从Queue中获取一个item，由NewNamedRateLimitingQueue创建而来，这里可
  // 能影响到日志刷新频率，如果Get一直夯住的话，那么对reconcilerHandler的调用会
  // 延迟
	obj, shutdown := c.Queue.Get()
	if shutdown {
		// Stop working
		return false
	}

	// We call Done here so the workqueue knows we have finished
	// processing this item. We also must remember to call Forget if we
	// do not want this work item being re-queued. For example, we do
	// not call Forget if a transient error occurs, instead the item is
	// put back on the workqueue and attempted again after a back-off
	// period.
	defer c.Queue.Done(obj)

	return c.reconcileHandler(obj)
}

func (c *Controller) reconcileHandler(obj interface{}) bool {
	// Update metrics after processing each item
	reconcileStartTS := time.Now()
	defer func() {
		c.updateMetrics(time.Since(reconcileStartTS))
	}()

	var req reconcile.Request
	var ok bool
	if req, ok = obj.(reconcile.Request); !ok {
		// As the item in the workqueue is actually invalid, we call
		// Forget here else we'd go into a loop of attempting to
		// process a work item that is invalid.
		c.Queue.Forget(obj)
		log.Error(nil, "Queue item was not a Request",
			"controller", c.Name, "type", fmt.Sprintf("%T", obj), "value", obj)
		// Return true, don't take a break
		return true
	}
	// RunInformersAndControllers the syncHandler, passing it the namespace/Name string of the
	// resource to be synced.
  // 在controller-runtime启动中分析过，c.Do被赋值为reconciler，所以
  // c.Do.Reconcile调用的便是reconciler.Reconcile
  // Reconcile函数是处理Event的核心逻辑，下面的判断分支即是根据Reconcile的返
  // 回是否含有错误进行不同的事件请求重入队
	if result, err := c.Do.Reconcile(req); err != nil {
    // 错误不为空，将这个事件请求重新加入到限速队列中，可能会影响到日志刷新的频率
		c.Queue.AddRateLimited(req)
		log.Error(err, "Reconciler error", "controller", c.Name, "request", req)
		ctrlmetrics.ReconcileErrors.WithLabelValues(c.Name).Inc()
		ctrlmetrics.ReconcileTotal.WithLabelValues(c.Name, "error").Inc()
    // 在错误不为空的情况下，返回false，会导致间接调用者processNextWorkItem的
    // 返回值为false，再往上追溯，会导致c.worker退出，wait.Until会间隔
    // JitterPeriod(1s)后重新调度，这可以解释日志中的前面的几条日志的间隔为1s，但
    // 是无法解释后面日志的时间间隔的递增
		return false
	} else if result.RequeueAfter > 0 {
		// The result.RequeueAfter request will be lost, if it is returned
		// along with a non-nil error. But this is intended as
		// We need to drive to stable reconcile loops before queuing due
		// to result.RequestAfter
    
    // 错误为空，但是设置了RequeueAfter的话，会将现有的事件请求作为一个新的请
    // 求在RequeueAfter后冲洗加入队列，作为新的请求的意思是清除这个请求已在队
    // 列中保存的其他数据
		c.Queue.Forget(obj)
		c.Queue.AddAfter(req, result.RequeueAfter)
		ctrlmetrics.ReconcileTotal.WithLabelValues(c.Name, "requeue_after").Inc()
    // 返回值为true，processNextWorkItem的返回一直为true，所以不会触发
    // wait.Until重新调度c.worker，因此如果有日志的话，预测日志的前几条应该不是按
    // 照1s间隔打印的，而是按照其他规则
		return true
	} else if result.Requeue {
    // 在错误为空且RequeueAfter不大于0的情况下，如果设置Requeue为true，那
    // 么仅将请求重新加入限速队列
		c.Queue.AddRateLimited(req)
		ctrlmetrics.ReconcileTotal.WithLabelValues(c.Name, "requeue").Inc()
    // 不会触发wait.Until的重新调度，所以日志应该也是规则的
		return true
	}

	// Finally, if no error occurs we Forget this item so it does not
	// get queued again until another change happens.
  // 上述判断分支都失败时，即错误为空，result也无其他设置时，认为请求被正确处理，所
	// 以直接在队里中清除该请求后返回，可以继续进行下一个请求的处理
	c.Queue.Forget(obj)

	// TODO(directxman12): What does 1 mean?  Do we want level constants?  Do we want levels at all?
	log.V(1).Info("Successfully Reconciled", "controller", c.Name, "request", req)

	ctrlmetrics.ReconcileTotal.WithLabelValues(c.Name, "success").Inc()
	// Return true, don't take a break
	return true
}

worker内部的最终逻辑回到了Reconcile方法，也即是需要在controllers/xxx_controller.go:40中的Reconcile()自定义逻辑的方法，按照自定义的逻辑运行。

参考：https://blog.csdn.net/ywq935/article/details/106311583

参考：https://blog.51cto.com/u_15072904/2615605

参考：https://book.kubebuilder.io/quick-start.html

Magiceses

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
纯干货整理，一文看懂 kubebuilder 安装，部署以及源码controller-runtime等库分析

快速开始构建go环境curl -o /tmp/go1.13.14.linux-amd64.tar.gz https://dl.google.com/go/go1.13.14.linux-amd64.tar.gzrm -rf /usr/local/go/ && tar -C /usr/local/ -zxf /tmp/go1.13.14.linux-amd64.tar.gzcat <<"EOF" | tee -a ~/.bashrc# add by chengql st
复制链接

扫一扫