文章目录
前言
本篇文章主要讲解了如果使用kubebuilder构建Operator项目。
一、环境准备
1.go环境
go环境搭建可以参考笔者之前的笔记macOS搭建go环境及VSCode安装使用教程
sunxi@sunxideMacBook-Pro vagrant-provisioning % go version
go version go1.16.5 darwin/amd64
2.kubebuilder
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/
3.kustomize
brew install kustomize
4.k8s集群
k8s集群搭建可以参考笔者之前的笔记Mac上自动化搭建k8s集群
[vagrant@kmaster ~]$ kubectl get no
NAME STATUS ROLES AGE VERSION
kmaster.example.com Ready control-plane,master 143d v1.20.2
kworker1.example.com Ready <none> 143d v1.20.2
kworker2.example.com Ready <none> 143d v1.20.2
二、kubebuilder demo
1.创建一个project
在$GOPATH/src下创建项目:
mkdir kubebuilder-demo
cd kubebuilder-demo
2.初始化项目
(1)在项目目录下执行以下命令,命令执行完成后该目录下会生成一个go.mod文件
go mod init my.domain
sunxi@sunxideMacBook-Pro kubebuilder-demo % go mod init my.domain
go: creating new go.mod: module my.domain
sunxi@sunxideMacBook-Pro kubebuilder-demo % ls -l
total 8
-rw-r--r-- 1 sunxi staff 26 6 29 10:21 go.mod
(2)在项目目录下执行以下命令完成项目的初始化
kubebuilder init --domain example.com
sunxi@sunxideMacBook-Pro kubebuilder-demo % kubebuilder init --domain example.com
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.8.3
go: downloading google.golang.org/appengine v1.6.6
go: downloading github.com/evanphx/json-patch v0.5.2
go: downloading k8s.io/client-go v1.5.2
Update dependencies:
$ go mod tidy
go: downloading github.com/stretchr/testify v1.6.1
go: downloading go.uber.org/goleak v1.1.10
go: downloading golang.org/x/lint v0.0.0-20200302205851-738671d3881b
go: downloading honnef.co/go/tools v0.0.1-2020.1.3
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.1
go: downloading gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15
go: downloading go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/kr/pretty v0.2.0
go: downloading golang.org/x/tools v0.0.0-20200616133436-c1934b75d054
go: downloading github.com/kr/text v0.1.0
Next: define a resource with:
$ kubebuilder create api
执行完该命令后可以看到在项目目录下多了很多目录及文件具体如下,该命令主要是拉取依赖代码库、生成代码框架以及Dockerfile、Makefile等工具文件
sunxi@sunxideMacBook-Pro kubebuilder-demo % ls -l
total 184
-rw------- 1 sunxi staff 776 6 29 10:38 Dockerfile
-rw------- 1 sunxi staff 4443 6 29 10:38 Makefile
-rw------- 1 sunxi staff 110 6 29 10:38 PROJECT
drwx------ 6 sunxi staff 192 6 29 10:38 config
-rw-r--r-- 1 sunxi staff 133 6 29 10:39 go.mod
-rw-r--r-- 1 sunxi staff 69190 6 29 10:39 go.sum
drwx------ 3 sunxi staff 96 6 29 10:38 hack
-rw------- 1 sunxi staff 2782 6 29 10:38 main.go
注意:这里可以通过help来查看其参数含义:
sunxi@sunxideMacBook-Pro kubebuilder-demo % kubebuilder init --help
Initialize a new project including the following files:
- a "go.mod" with project dependencies
- a "PROJECT" file that stores project configuration
- a "Makefile" with several useful make targets for the project
- several YAML files for project deployment under the "config" directory
- a "main.go" file that creates the manager that will run the project controllers
Usage:
kubebuilder init [flags]
Examples:
# Initialize a new project with your domain and name in copyright
kubebuilder init --plugins go/v3 --domain example.org --owner "Your name"
# Initialize a new project defining an specific project version
kubebuilder init --plugins go/v3 --project-version 3
Flags:
--component-config create a versioned ComponentConfig file, may be 'true' or 'false'
--domain string domain for groups (default "my.domain")
--fetch-deps ensure dependencies are downloaded (default true)
-h, --help help for init
--license string license to use to boilerplate, may be one of 'apache2', 'none' (default "apache2")
--owner string owner to add to the copyright
--project-name string name of this project
--project-version string project version (default "3")
--repo string name to use for go module (e.g., github.com/user/repo), defaults to the go package of the current working directory.
--skip-go-version-check if specified, skip checking the Go version
Global Flags:
--plugins strings plugin keys to be used for this subcommand execution
(3)执行以下命令创建CRD api
kubebuilder create api --group webapp --version v1 --kind Frigate
这里需要注意以下几点:
- group、version与kind这三个属性用来标识一个CRD
- 在创建的过程中会出现选择:是否生成Resource以及Controller,这里输入y即可
sunxi@sunxideMacBook-Pro kubebuilder-demo % kubebuilder create api --group webapp --version v1 --kind Frigate
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/frigate_types.go
controllers/frigate_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.4.1
go: downloading sigs.k8s.io/controller-tools v0.4.1
go: downloading github.com/spf13/cobra v1.0.0
go: downloading golang.org/x/tools v0.0.0-20200616195046-dc31b401abb5
go: downloading github.com/fatih/color v1.7.0
go: downloading k8s.io/api v0.18.2
go: downloading k8s.io/apimachinery v0.18.2
go: downloading gopkg.in/yaml.v3 v3.0.0-20190905181640-827449938966
go: downloading k8s.io/apiextensions-apiserver v0.18.2
go: downloading github.com/gobuffalo/flect v0.2.0
go: downloading gopkg.in/yaml.v2 v2.2.8
go: downloading github.com/mattn/go-colorable v0.1.2
go: downloading github.com/mattn/go-isatty v0.0.8
go: downloading k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89
go: downloading k8s.io/klog v1.0.0
go: downloading sigs.k8s.io/structured-merge-diff/v3 v3.0.0
go: downloading golang.org/x/net v0.0.0-20200226121028-0de0cce0169b
go: downloading golang.org/x/sys v0.0.0-20191022100944-742c48ecaeb7
go: downloading github.com/json-iterator/go v1.1.8
go: downloading golang.org/x/text v0.3.2
go get: added sigs.k8s.io/controller-tools v0.4.1
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
sunxi@sunxideMacBook-Pro kubebuilder-demo %
这一步执行完成后可以发现项目的目录下面多了api、bin、controllers目录
sunxi@sunxideMacBook-Pro kubebuilder-demo % ls -l
total 184
-rw------- 1 sunxi staff 776 6 29 10:38 Dockerfile
-rw------- 1 sunxi staff 4443 6 29 10:38 Makefile
-rw------- 1 sunxi staff 280 6 29 10:59 PROJECT
drwx------ 3 sunxi staff 96 6 29 10:59 api
drwxr-xr-x 3 sunxi staff 96 6 29 11:01 bin
drwx------ 8 sunxi staff 256 6 29 10:59 config
drwx------ 4 sunxi staff 128 6 29 10:59 controllers
-rw-r--r-- 1 sunxi staff 197 6 29 10:59 go.mod
-rw-r--r-- 1 sunxi staff 69190 6 29 10:39 go.sum
drwx------ 3 sunxi staff 96 6 29 10:38 hack
-rw------- 1 sunxi staff 3116 6 29 10:59 main.go
注意:这里可以通过help来查看其参数含义:
sunxi@sunxideMacBook-Pro kubebuilder-demo % kubebuilder create api --help
Scaffold a Kubernetes API by writing a Resource definition and/or a Controller.
If information about whether the resource and controller should be scaffolded
was not explicitly provided, it will prompt the user if they should be.
After the scaffold is written, the dependencies will be updated and
make generate will be run.
Usage:
kubebuilder create api [flags]
Examples:
# Create a frigates API with Group: ship, Version: v1beta1 and Kind: Frigate
kubebuilder create api --group ship --version v1beta1 --kind Frigate
# Edit the API Scheme
nano api/v1beta1/frigate_types.go
# Edit the Controller
nano controllers/frigate/frigate_controller.go
# Edit the Controller Test
nano controllers/frigate/frigate_controller_test.go
# Install CRDs into the Kubernetes cluster using kubectl apply
make install
# Regenerate code and run against the Kubernetes cluster configured by ~/.kube/config
make run
Flags:
--controller if set, generate the controller without prompting the user (default true)
--crd-version string version of CustomResourceDefinition to scaffold. Options: [v1, v1beta1] (default "v1")
--force attempt to create resource even if it already exists
--group string resource Group
-h, --help help for api
--kind string resource Kind
--make make generate if true, run make generate after generating files (default true)
--namespaced resource is namespaced (default true)
--plural string resource irregular plural form
--resource if set, generate the resource without prompting the user (default true)
--version string resource Version
Global Flags:
--plugins strings plugin keys to be used for this subcommand execution
3.在宿主机上安装kubectl
在宿主机上安装kubectl便于在集群外访问k8s
(1)安装
curl -LO http://kubernetes.oss-cn-hangzhou.aliyuncs.com/kubernetes-release/release/v1.20.2/bin/darwin/amd64/kubectl
# 修改为可执行的文件
chmod +x ./kubectl
# 将命令放置于环境变量中
mv ./kubectl /usr/local/bin/kubectl
(2)配置
在宿主机的主目录下创建.kube目录,并在该目录下创建config文件,将k8s集群中对应的config文件内容复制到新建的config文件即可。
然后在宿主机上执行以下命令验证:
sunxi@sunxideMacBook-Pro .kube % kubectl cluster-info
Kubernetes control plane is running at https://172.42.42.100:6443
KubeDNS is running at https://172.42.42.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
sunxi@sunxideMacBook-Pro .kube %
4.安装CRD并部署controller
执行以下命令将CRD安装到k8s集群:
make install
sunxi@sunxideMacBook-Pro kubebuilder-demo % make install
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
go: downloading sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go: downloading sigs.k8s.io/kustomize/api v0.6.5
go: downloading k8s.io/client-go v0.18.10
go: downloading sigs.k8s.io/kustomize/cmd/config v0.8.5
go: downloading k8s.io/apimachinery v0.18.10
go: downloading sigs.k8s.io/kustomize/kyaml v0.9.4
go: downloading github.com/yujunz/go-getter v1.4.1-lite
go: downloading github.com/go-errors/errors v1.0.1
go: downloading github.com/go-openapi/spec v0.19.5
go: downloading github.com/olekukonko/tablewriter v0.0.4
go: downloading k8s.io/kube-openapi v0.0.0-20200410145947-61e04a5be9a6
go: downloading k8s.io/api v0.18.10
go: downloading sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e
go: downloading github.com/hashicorp/go-multierror v1.1.0
go: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
go: downloading github.com/hashicorp/go-cleanhttp v0.5.0
go: downloading github.com/hashicorp/go-safetemp v1.0.0
go: downloading github.com/hashicorp/go-version v1.1.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/mitchellh/go-testing-interface v1.0.0
go: downloading github.com/ulikunitz/xz v0.5.5
go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go: downloading github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
go: downloading github.com/go-openapi/strfmt v0.19.5
go: downloading github.com/go-openapi/validate v0.19.8
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading github.com/mattn/go-runewidth v0.0.7
go: downloading github.com/go-openapi/jsonpointer v0.19.3
go: downloading github.com/go-openapi/jsonreference v0.19.3
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/go-openapi/swag v0.19.5
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/googleapis/gnostic v0.1.0
go: downloading golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go: downloading github.com/Azure/go-autorest/autorest v0.9.0
go: downloading github.com/gophercloud/gophercloud v0.1.0
go: downloading github.com/qri-io/starlib v0.4.2-0.20200213133954-ff2e8cd5ef8d
go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
go: downloading github.com/Azure/go-autorest/autorest/adal v0.5.0
go: downloading golang.org/x/net v0.0.0-20200625001655-4c5254603344
go: downloading github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633
go: downloading github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: downloading github.com/go-openapi/errors v0.19.2
go: downloading github.com/mitchellh/mapstructure v1.1.2
go: downloading go.mongodb.org/mongo-driver v1.1.2
go: downloading github.com/PuerkitoBio/purell v1.1.1
go: downloading github.com/go-openapi/analysis v0.19.5
go: downloading github.com/go-openapi/loads v0.19.4
go: downloading github.com/go-openapi/runtime v0.19.4
go: downloading github.com/mailru/easyjson v0.7.0
go: downloading github.com/golang/protobuf v1.3.2
go: downloading cloud.google.com/go v0.38.0
go: downloading google.golang.org/appengine v1.5.0
go: downloading github.com/Azure/go-autorest/tracing v0.5.0
go: downloading github.com/Azure/go-autorest/logger v0.1.0
go: downloading github.com/Azure/go-autorest/autorest/date v0.1.0
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go: downloading golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: downloading github.com/go-stack/stack v1.8.0
go: downloading golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
go: downloading github.com/dgrijalva/jwt-go v1.0.2
go: downloading github.com/emicklei/go-restful v1.1.3
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/frigates.webapp.example.com created
验证安装是否成功:
sunxi@sunxideMacBook-Pro kubebuilder-demo % kubectl get crd
NAME CREATED AT
frigates.webapp.example.com 2021-06-29T08:24:12Z
部署controller:
- 本地部署
sunxi@bogon kubebuilder-demo % make run
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
2021-06-29T20:33:52.667+0800 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2021-06-29T20:33:52.668+0800 INFO setup starting manager
2021-06-29T20:33:52.668+0800 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2021-06-29T20:33:52.669+0800 INFO controller-runtime.manager.controller.frigate Starting EventSource {"reconciler group": "webapp.example.com", "reconciler kind": "Frigate", "source": "kind source: /, Kind="}
2021-06-29T20:33:52.771+0800 INFO controller-runtime.manager.controller.frigate Starting Controller {"reconciler group": "webapp.example.com", "reconciler kind": "Frigate"}
2021-06-29T20:33:52.771+0800 INFO controller-runtime.manager.controller.frigate Starting workers {"reconciler group": "webapp.example.com", "reconciler kind": "Frigate", "worker count": 1}
- 远程部署
远程部署相关命令如下,这里笔者没有验证构建docker镜像,笔者本地有搭建好的 k8s集群所以直接执行make deploy是没有问题的
make docker-build docker-push IMG=<some-registry>/<project-name>:tag
make deploy IMG=<some-registry>/<project-name>:tag
5.创建CR
Kubebuilder 在初始化项目的时候已生成了示例 CR,具体位置是:$GOPATH/src/kubebuilder-demo/config/samples
sunxi@bogon samples % pwd
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/config/samples
sunxi@bogon samples %
sunxi@bogon samples % ls -l
total 8
-rw------- 1 sunxi staff 118 6 29 10:59 webapp_v1_frigate.yaml
sunxi@bogon samples %
sunxi@bogon samples % cat webapp_v1_frigate.yaml
apiVersion: webapp.example.com/v1
kind: Frigate
metadata:
name: frigate-sample
spec:
# Add fields here
foo: bar
sunxi@bogon samples %
执行以下命令部署CR:
sunxi@bogon samples % kubectl apply -f webapp_v1_frigate.yaml
frigate.webapp.example.com/frigate-sample created
sunxi@bogon samples %
sunxi@bogon samples % kubectl get Frigate
NAME AGE
frigate-sample 52s
查看新建的CR:
sunxi@bogon samples % kubectl get frigate.webapp.example.com frigate-sample -o yaml
apiVersion: webapp.example.com/v1
kind: Frigate
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"webapp.example.com/v1","kind":"Frigate","metadata":{"annotations":{},"name":"frigate-sample","namespace":"default"},"spec":{"foo":"bar"}}
creationTimestamp: "2021-06-29T08:46:10Z"
generation: 1
managedFields:
- apiVersion: webapp.example.com/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:foo: {}
manager: kubectl-client-side-apply
operation: Update
time: "2021-06-29T08:46:10Z"
name: frigate-sample
namespace: default
resourceVersion: "42185"
uid: bfbc73a2-436e-4dce-9b09-8e5c2787d028
spec:
foo: bar
6.删除CR及卸载CRD
- 删除CR
sunxi@bogon samples % kubectl delete frigate.webapp.example.com frigate-sample
frigate.webapp.example.com "frigate-sample" deleted
- 卸载CRD
sunxi@bogon kubebuilder-demo % make uninstall
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/Users/sunxi/Documents/goCode/src/kubebuilder-demo/bin/kustomize build config/crd | kubectl delete -f -
customresourcedefinition.apiextensions.k8s.io "frigates.webapp.example.com" deleted
7.修改CRD定义及核心逻辑
这里笔者简单的修改了项目中frigate_types.go、frigate_controller.go以及main.go并进行了测试,项目也放在了GitHub上,项目地址:kubebuilder-demo
8.项目目录总结
目录 | 含义 |
---|---|
Dockerfile | 构建docker镜像的Dockerfile |
Makefile | make编译文件 |
PROJECT | 项目元数据 |
api/v1/xxx_types.go | 自定义的CRD结构 |
api/v1/groupversion_info.go | GVK信息 |
api/v1/zz_generated.deepcopy.go | 对象复制 |
config | kubectl apply的资源 |
controllers/xxx_controller.go | CRD controller核心逻辑 |
controllers/suite_test.go | 测试 |
main.go | 项目入口 |
三、参考资料
https://book.kubebuilder.io/quick-start.html
https://blog.csdn.net/weixin_42072280/article/details/112857621
阿里云原生技术公开课:
- kubernetes API编程范式
https://edu.aliyun.com/lesson_1651_13094?spm=5176.10731542.0.0.5bd77abdEPE5ti#_13094 - kubernetes API编程利器-Operator和OperatorFramework
https://edu.aliyun.com/lesson_1651_13095?spm=5176.10731542.0.0.638a7abdJk2d12#_13095