kubebuilder创建operator

---

前言

本篇文章主要讲解了如果使用kubebuilder构建Operator项目。

---

一、环境准备

1.go环境
go环境搭建可以参考笔者之前的笔记macOS搭建go环境及VSCode安装使用教程

sunxi@sunxideMacBook-Pro vagrant-provisioning % go version
go version go1.16.5 darwin/amd64

2.kubebuilder

curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)

chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

3.kustomize

brew install kustomize

4.k8s集群
k8s集群搭建可以参考笔者之前的笔记Mac上自动化搭建k8s集群

[vagrant@kmaster ~]$ kubectl get no
NAME                   STATUS   ROLES                  AGE    VERSION
kmaster.example.com    Ready    control-plane,master   143d   v1.20.2
kworker1.example.com   Ready    <none>                 143d   v1.20.2
kworker2.example.com   Ready    <none>                 143d   v1.20.2

二、kubebuilder demo

1.创建一个project

在$GOPATH/src下创建项目：

mkdir kubebuilder-demo
cd kubebuilder-demo 

2.初始化项目

（1）在项目目录下执行以下命令，命令执行完成后该目录下会生成一个go.mod文件

go mod init my.domain

sunxi@sunxideMacBook-Pro kubebuilder-demo % go mod init my.domain
go: creating new go.mod: module my.domain
sunxi@sunxideMacBook-Pro kubebuilder-demo % ls -l
total 8
-rw-r--r--  1 sunxi  staff  26  6 29 10:21 go.mod

（2）在项目目录下执行以下命令完成项目的初始化

kubebuilder init --domain example.com

sunxi@sunxideMacBook-Pro kubebuilder-demo % kubebuilder init --domain example.com
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.8.3
go: downloading google.golang.org/appengine v1.6.6
go: downloading github.com/evanphx/json-patch v0.5.2
go: downloading k8s.io/client-go v1.5.2
Update dependencies:
$ go mod tidy
go: downloading github.com/stretchr/testify v1.6.1
go: downloading go.uber.org/goleak v1.1.10
go: downloading golang.org/x/lint v0.0.0-20200302205851-738671d3881b
go: downloading honnef.co/go/tools v0.0.1-2020.1.3
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.1
go: downloading gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15
go: downloading go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/kr/pretty v0.2.0
go: downloading golang.org/x/tools v0.0.0-20200616133436-c1934b75d054
go: downloading github.com/kr/text v0.1.0
Next: define a resource with:
$ kubebuilder create api

执行完该命令后可以看到在项目目录下多了很多目录及文件具体如下，该命令主要是拉取依赖代码库、生成代码框架以及Dockerfile、Makefile等工具文件

sunxi@sunxideMacBook-Pro kubebuilder-demo % ls -l
total 184
-rw-------  1 sunxi  staff    776  6 29 10:38 Dockerfile
-rw-------  1 sunxi  staff   4443  6 29 10:38 Makefile
-rw-------  1 sunxi  staff    110  6 29 10:38 PROJECT
drwx------  6 sunxi  staff    192  6 29 10:38 config
-rw-r--r--  1 sunxi  staff    133  6 29 10:39 go.mod
-rw-r--r--  1 sunxi  staff  69190  6 29 10:39 go.sum
drwx------  3 sunxi  staff     96  6 29 10:38 hack
-rw-------  1 sunxi  staff   2782  6 29 10:38 main.go

注意：这里可以通过help来查看其参数含义：

sunxi@sunxideMacBook-Pro kubebuilder-demo % kubebuilder init --help
Initialize a new project including the following files:
  - a "go.mod" with project dependencies
  - a "PROJECT" file that stores project configuration
  - a "Makefile" with several useful make targets for the project
  - several YAML files for project deployment under the "config" directory
  - a "main.go" file that creates the manager that will run the project controllers

Usage:
  kubebuilder init [flags]

Examples:
  # Initialize a new project with your domain and name in copyright
  kubebuilder init --plugins go/v3 --domain example.org --owner "Your name"

  # Initialize a new project defining an specific project version
  kubebuilder init --plugins go/v3 --project-version 3


Flags:
      --component-config         create a versioned ComponentConfig file, may be 'true' or 'false'
      --domain string            domain for groups (default "my.domain")
      --fetch-deps               ensure dependencies are downloaded (default true)
  -h, --help                     help for init
      --license string           license to use to boilerplate, may be one of 'apache2', 'none' (default "apache2")
      --owner string             owner to add to the copyright
      --project-name string      name of this project
      --project-version string   project version (default "3")
      --repo string              name to use for go module (e.g., github.com/user/repo), defaults to the go package of the current working directory.
      --skip-go-version-check    if specified, skip checking the Go version

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution

（3）执行以下命令创建CRD api

kubebuilder create api --group webapp --version v1 --kind Frigate

这里需要注意以下几点：

• group、version与kind这三个属性用来标识一个CRD
• 在创建的过程中会出现选择：是否生成Resource以及Controller，这里输入y即可

sunxi@sunxideMacBook-Pro kubebuilder-demo % kubebuilder create api --group webapp --version v1 --kind Frigate
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/frigate_types.go
controllers/frigate_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.4.1
go: downloading sigs.k8s.io/controller-tools v0.4.1
go: downloading github.com/spf13/cobra v1.0.0
go: downloading golang.org/x/tools v0.0.0-20200616195046-dc31b401abb5
go: downloading github.com/fatih/color v1.7.0
go: downloading k8s.io/api v0.18.2
go: downloading k8s.io/apimachinery v0.18.2
go: downloading gopkg.in/yaml.v3 v3.0.0-20190905181640-827449938966
go: downloading k8s.io/apiextensions-apiserver v0.18.2
go: downloading github.com/gobuffalo/flect v0.2.0
go: downloading gopkg.in/yaml.v2 v2.2.8
go: downloading github.com/mattn/go-colorable v0.1.2
go: downloading github.com/mattn/go-isatty v0.0.8
go: downloading k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89
go: downloading k8s.io/klog v1.0.0
go: downloading sigs.k8s.io/structured-merge-diff/v3 v3.0.0
go: downloading golang.org/x/net v0.0.0-20200226121028-0de0cce0169b
go: downloading golang.org/x/sys v0.0.0-20191022100944-742c48ecaeb7
go: downloading github.com/json-iterator/go v1.1.8
go: downloading golang.org/x/text v0.3.2
go get: added sigs.k8s.io/controller-tools v0.4.1
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
sunxi@sunxideMacBook-Pro kubebuilder-demo % 

这一步执行完成后可以发现项目的目录下面多了api、bin、controllers目录

sunxi@sunxideMacBook-Pro kubebuilder-demo % ls -l
total 184
-rw-------  1 sunxi  staff    776  6 29 10:38 Dockerfile
-rw-------  1 sunxi  staff   4443  6 29 10:38 Makefile
-rw-------  1 sunxi  staff    280  6 29 10:59 PROJECT
drwx------  3 sunxi  staff     96  6 29 10:59 api
drwxr-xr-x  3 sunxi  staff     96  6 29 11:01 bin
drwx------  8 sunxi  staff    256  6 29 10:59 config
drwx------  4 sunxi  staff    128  6 29 10:59 controllers
-rw-r--r--  1 sunxi  staff    197  6 29 10:59 go.mod
-rw-r--r--  1 sunxi  staff  69190  6 29 10:39 go.sum
drwx------  3 sunxi  staff     96  6 29 10:38 hack
-rw-------  1 sunxi  staff   3116  6 29 10:59 main.go

注意：这里可以通过help来查看其参数含义：

sunxi@sunxideMacBook-Pro kubebuilder-demo % kubebuilder create api --help
Scaffold a Kubernetes API by writing a Resource definition and/or a Controller.

If information about whether the resource and controller should be scaffolded
was not explicitly provided, it will prompt the user if they should be.

After the scaffold is written, the dependencies will be updated and
make generate will be run.

Usage:
  kubebuilder create api [flags]

Examples:
  # Create a frigates API with Group: ship, Version: v1beta1 and Kind: Frigate
  kubebuilder create api --group ship --version v1beta1 --kind Frigate

  # Edit the API Scheme
  nano api/v1beta1/frigate_types.go

  # Edit the Controller
  nano controllers/frigate/frigate_controller.go

  # Edit the Controller Test
  nano controllers/frigate/frigate_controller_test.go

  # Install CRDs into the Kubernetes cluster using kubectl apply
  make install

  # Regenerate code and run against the Kubernetes cluster configured by ~/.kube/config
  make run


Flags:
      --controller           if set, generate the controller without prompting the user (default true)
      --crd-version string   version of CustomResourceDefinition to scaffold. Options: [v1, v1beta1] (default "v1")
      --force                attempt to create resource even if it already exists
      --group string         resource Group
  -h, --help                 help for api
      --kind string          resource Kind
      --make make generate   if true, run make generate after generating files (default true)
      --namespaced           resource is namespaced (default true)
      --plural string        resource irregular plural form
      --resource             if set, generate the resource without prompting the user (default true)
      --version string       resource Version

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution

3.在宿主机上安装kubectl

在宿主机上安装kubectl便于在集群外访问k8s
(1)安装

curl -LO http://kubernetes.oss-cn-hangzhou.aliyuncs.com/kubernetes-release/release/v1.20.2/bin/darwin/amd64/kubectl
# 修改为可执行的文件 
chmod +x ./kubectl 
# 将命令放置于环境变量中 
mv ./kubectl /usr/local/bin/kubectl 

(2)配置
在宿主机的主目录下创建.kube目录，并在该目录下创建config文件，将k8s集群中对应的config文件内容复制到新建的config文件即可。
然后在宿主机上执行以下命令验证：

sunxi@sunxideMacBook-Pro .kube % kubectl cluster-info
Kubernetes control plane is running at https://172.42.42.100:6443
KubeDNS is running at https://172.42.42.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
sunxi@sunxideMacBook-Pro .kube % 

4.安装CRD并部署controller

执行以下命令将CRD安装到k8s集群：

make install

sunxi@sunxideMacBook-Pro kubebuilder-demo % make install
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
go: downloading sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go: downloading sigs.k8s.io/kustomize/api v0.6.5
go: downloading k8s.io/client-go v0.18.10
go: downloading sigs.k8s.io/kustomize/cmd/config v0.8.5
go: downloading k8s.io/apimachinery v0.18.10
go: downloading sigs.k8s.io/kustomize/kyaml v0.9.4
go: downloading github.com/yujunz/go-getter v1.4.1-lite
go: downloading github.com/go-errors/errors v1.0.1
go: downloading github.com/go-openapi/spec v0.19.5
go: downloading github.com/olekukonko/tablewriter v0.0.4
go: downloading k8s.io/kube-openapi v0.0.0-20200410145947-61e04a5be9a6
go: downloading k8s.io/api v0.18.10
go: downloading sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e
go: downloading github.com/hashicorp/go-multierror v1.1.0
go: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
go: downloading github.com/hashicorp/go-cleanhttp v0.5.0
go: downloading github.com/hashicorp/go-safetemp v1.0.0
go: downloading github.com/hashicorp/go-version v1.1.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/mitchellh/go-testing-interface v1.0.0
go: downloading github.com/ulikunitz/xz v0.5.5
go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go: downloading github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
go: downloading github.com/go-openapi/strfmt v0.19.5
go: downloading github.com/go-openapi/validate v0.19.8
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading github.com/mattn/go-runewidth v0.0.7
go: downloading github.com/go-openapi/jsonpointer v0.19.3
go: downloading github.com/go-openapi/jsonreference v0.19.3
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/go-openapi/swag v0.19.5
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/googleapis/gnostic v0.1.0
go: downloading golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go: downloading github.com/Azure/go-autorest/autorest v0.9.0
go: downloading github.com/gophercloud/gophercloud v0.1.0
go: downloading github.com/qri-io/starlib v0.4.2-0.20200213133954-ff2e8cd5ef8d
go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
go: downloading github.com/Azure/go-autorest/autorest/adal v0.5.0
go: downloading golang.org/x/net v0.0.0-20200625001655-4c5254603344
go: downloading github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633
go: downloading github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: downloading github.com/go-openapi/errors v0.19.2
go: downloading github.com/mitchellh/mapstructure v1.1.2
go: downloading go.mongodb.org/mongo-driver v1.1.2
go: downloading github.com/PuerkitoBio/purell v1.1.1
go: downloading github.com/go-openapi/analysis v0.19.5
go: downloading github.com/go-openapi/loads v0.19.4
go: downloading github.com/go-openapi/runtime v0.19.4
go: downloading github.com/mailru/easyjson v0.7.0
go: downloading github.com/golang/protobuf v1.3.2
go: downloading cloud.google.com/go v0.38.0
go: downloading google.golang.org/appengine v1.5.0
go: downloading github.com/Azure/go-autorest/tracing v0.5.0
go: downloading github.com/Azure/go-autorest/logger v0.1.0
go: downloading github.com/Azure/go-autorest/autorest/date v0.1.0
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go: downloading golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: downloading github.com/go-stack/stack v1.8.0
go: downloading golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
go: downloading github.com/dgrijalva/jwt-go v1.0.2
go: downloading github.com/emicklei/go-restful v1.1.3
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/frigates.webapp.example.com created

验证安装是否成功：

sunxi@sunxideMacBook-Pro kubebuilder-demo % kubectl get crd
NAME                          CREATED AT
frigates.webapp.example.com   2021-06-29T08:24:12Z

部署controller：

• 本地部署

sunxi@bogon kubebuilder-demo % make run
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
2021-06-29T20:33:52.667+0800	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-06-29T20:33:52.668+0800	INFO	setup	starting manager
2021-06-29T20:33:52.668+0800	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-06-29T20:33:52.669+0800	INFO	controller-runtime.manager.controller.frigate	Starting EventSource	{"reconciler group": "webapp.example.com", "reconciler kind": "Frigate", "source": "kind source: /, Kind="}
2021-06-29T20:33:52.771+0800	INFO	controller-runtime.manager.controller.frigate	Starting Controller	{"reconciler group": "webapp.example.com", "reconciler kind": "Frigate"}
2021-06-29T20:33:52.771+0800	INFO	controller-runtime.manager.controller.frigate	Starting workers	{"reconciler group": "webapp.example.com", "reconciler kind": "Frigate", "worker count": 1}

• 远程部署
  远程部署相关命令如下，这里笔者没有验证构建docker镜像，笔者本地有搭建好的 k8s集群所以直接执行make deploy是没有问题的

make docker-build docker-push IMG=<some-registry>/<project-name>:tag
make deploy IMG=<some-registry>/<project-name>:tag

5.创建CR

Kubebuilder 在初始化项目的时候已生成了示例 CR，具体位置是：$GOPATH/src/kubebuilder-demo/config/samples

sunxi@bogon samples % pwd
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/config/samples
sunxi@bogon samples % 
sunxi@bogon samples % ls -l
total 8
-rw-------  1 sunxi  staff  118  6 29 10:59 webapp_v1_frigate.yaml
sunxi@bogon samples % 
sunxi@bogon samples % cat webapp_v1_frigate.yaml 
apiVersion: webapp.example.com/v1
kind: Frigate
metadata:
  name: frigate-sample
spec:
  # Add fields here
  foo: bar
sunxi@bogon samples % 

执行以下命令部署CR：

sunxi@bogon samples % kubectl apply -f webapp_v1_frigate.yaml 
frigate.webapp.example.com/frigate-sample created
sunxi@bogon samples % 

sunxi@bogon samples % kubectl get Frigate
NAME             AGE
frigate-sample   52s

查看新建的CR：

sunxi@bogon samples % kubectl get frigate.webapp.example.com frigate-sample -o yaml
apiVersion: webapp.example.com/v1
kind: Frigate
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"webapp.example.com/v1","kind":"Frigate","metadata":{"annotations":{},"name":"frigate-sample","namespace":"default"},"spec":{"foo":"bar"}}
  creationTimestamp: "2021-06-29T08:46:10Z"
  generation: 1
  managedFields:
 - apiVersion: webapp.example.com/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        .: {}
        f:foo: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-06-29T08:46:10Z"
  name: frigate-sample
  namespace: default
  resourceVersion: "42185"
  uid: bfbc73a2-436e-4dce-9b09-8e5c2787d028
spec:
  foo: bar

6.删除CR及卸载CRD

• 删除CR

sunxi@bogon samples % kubectl delete frigate.webapp.example.com frigate-sample
frigate.webapp.example.com "frigate-sample" deleted

• 卸载CRD

sunxi@bogon kubebuilder-demo % make uninstall
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/kustomize build config/crd | kubectl delete -f -
customresourcedefinition.apiextensions.k8s.io "frigates.webapp.example.com" deleted

7.修改CRD定义及核心逻辑

这里笔者简单的修改了项目中frigate_types.go、frigate_controller.go以及main.go并进行了测试，项目也放在了GitHub上，项目地址：kubebuilder-demo

8.项目目录总结

目录	含义
Dockerfile	构建docker镜像的Dockerfile
Makefile	make编译文件
PROJECT	项目元数据
api/v1/xxx_types.go	自定义的CRD结构
api/v1/groupversion_info.go	GVK信息
api/v1/zz_generated.deepcopy.go	对象复制
config	kubectl apply的资源
controllers/xxx_controller.go	CRD controller核心逻辑
controllers/suite_test.go	测试
main.go	项目入口

三、参考资料

https://book.kubebuilder.io/quick-start.html
https://blog.csdn.net/weixin_42072280/article/details/112857621
阿里云原生技术公开课：

• kubernetes API编程范式
  https://edu.aliyun.com/lesson_1651_13094?spm=5176.10731542.0.0.5bd77abdEPE5ti#_13094
• kubernetes API编程利器-Operator和OperatorFramework
  https://edu.aliyun.com/lesson_1651_13095?spm=5176.10731542.0.0.638a7abdJk2d12#_13095