- 本文是 Kubernetes operator学习 系列第四篇,主要对 使用 controller-tools 进行 CRD 自动代码生成进行学习
- Kubernetes operator学习系列 快捷链接
1.controller-tools 简介
1.1.code-generator自动生成代码存在的问题
- Kubernetes operator(三)code-generator 篇 中,我们提到,code-generator编写CRD控制器有两个问题:
- 问题一:需要手动编写CRD的yaml,无法自动生成
- 问题二:types.go文件全部内容都需要我们手写,无法自动生成框架
- 这部分工作量其实也是挺大的,kubernetes提供了一个工具 controller-tools,可以对这部分内容也进行代码自动生成
1.2.controller-tools是什么
1.2.1.kubernetes-sigs 项目是什么
- kubernetes-sigs 是一个由 Kubernetes 社区维护的 GitHub 组织,其中包含了许多与 Kubernetes 相关的项目,这些项目通常是为 Kubernetes 生态系统开发的,用于提供各种功能和工具。
- 一些 kubernetes-sigs 组织中的流行项目包括:
- kustomize:一种用于 Kubernetes 部署的配置管理工具,可以通过 YAML 声明文件对 Kubernetes 对象进行自定义,并且支持多环境部署(例如 dev、stage、prod)。
- kubebuilder:一种用于构建 Kubernetes API 的 SDK 工具,可以帮助开发者快速构建和测试 Kubernetes 的自定义控制器。
- cluster-api:一种 Kubernetes 的 API 扩展,用于管理 Kubernetes 集群的生命周期,包括创建、扩容和删除。它允许开发者使用 Kubernetes 的声明性 API 来管理整个集群的生命周期。
- kubefed:用于跨 Kubernetes 集群联邦的控制平面。它提供了一种将多个 Kubernetes 集群组合成一个统一的逻辑实体的方法,同时保持每个集群的独立性。
- controller-tools:用于简化 Kubernetes 控制器的开发,提供了一组工具来生成和更新 Kubernetes API 对象的代码,以及构建自定义控制器所需的代码框架。
1.2.2.controller-tools是什么
- controller-tools其实是一个由 Kubernetes 社区维护的项目,用于简化 Kubernetes 控制器的开发。其中提供了一组工具来生成和更新 Kubernetes API 对象的代码,以及构建自定义控制器所需的代码框架。
- controller-tools 的github地址:https://github.com/kubernetes-sigs/controller-tools
1.2.3.controller-tools 包含哪些工具
- 在controller-tools源码的cmd目录下,可以看到包含三个工具
controller-gen
:用于生成zz_xxx.deepcopy.go
文件以及crd
文件【kubebuilder也是通过这个工具生成crd的相关框架的】type-scaffold
:用于生成所需的types.go
文件helpgen
:用于生成针对 Kubernetes API 对象的代码文档,可以包括 API 对象的字段、标签和注释等信息
2.controller-tools 使用过程
2.1.controller-tools 的 安装
-
controller-tools 的 github 地址:https://github.com/kubernetes-sigs/controller-tools.git,克隆代码
git clone https://github.com/kubernetes-sigs/controller-tools.git
-
将分支切换到 v0.9.0 的tag上
git checkout v0.9.0
-
编译项目,安装代码生成工具,这里我们只安装需要的2个工具
- controller-gen工具:生成
deepcopy方法
文件 +crd
文件 - type-scaffold工具:生成
types.go
文件
cd controller-tools # linux下安装,执行这一条即可 go install ./cmd/{controller-gen,type-scaffold} # windows下安装,需要执行两条命令 go install ./cmd/controller-gen go install ./cmd/type-scaffold
- controller-gen工具:生成
-
查看安装结果
- Linux下,在 GOPATH 下的 bin 目录下,会出现我们安装的工具 controller-gen、type-scaffold
- Windows下,在GOPATH/bin/windows_amd64下,有这两个工具
- Linux下,在 GOPATH 下的 bin 目录下,会出现我们安装的工具 controller-gen、type-scaffold
-
检查环境变量设置是否成功(Linux)
- 打开终端,执行 type-scaffold --help,如果报错:
[root@master zgy]# type-scaffold --help -bash: type-scaffold: 未找到命令
- 说明环境变量没有设置成功,需要将 gopath/bin 加入PATH
vim ~/.bashrc # 在~/.bashrc文件的末尾,加上这么一句 export PATH="$PATH:$GOPATH/bin" # 然后,source一下 source ~/.bashrc
- 再执行 type-scaffold --help,就成功了
[root@master zgy]# type-scaffold --help Quickly scaffold out the structure of a type for a Kubernetes kind and associated types. Produces: - a root type with appropriate metadata fields - Spec and Status types - a list type Also applies the appropriate comments to generate the code required to conform to runtime.Object. Usage: type-scaffold [flags] Examples: # Generate types for a Kind called Foo with a resource called foos type-scaffold --kind Foo # Generate types for a Kind called Bar with a resource of foobars type-scaffold --kind Bar --resource foobars Flags: -h, --help help for type-scaffold --kind string The kind of the typescaffold being scaffolded. --namespaced Whether or not the given resource is namespaced. (default true) --resource string The resource of the typescaffold being scaffolded (defaults to a lower-case, plural version of kind).
- 打开终端,执行 type-scaffold --help,如果报错:
2.2.type-scaffold 的使用方法
- type-scaffold 常用命令
type-scaffold --kind <Kind> [flags] type-scaffold --help
--kind
:参数用于指定要创建的资源类型(例如 Application)
2.3.controller-gen 的使用方法
- controller-gen 常用命令
# 生成 CRD 文件,并将生成的文件输出到 config/crds 目录中 controller-gen crd paths=./... output:crd:dir=config/crds # 生成与对象相关的代码,通常是指生成控制器相关的代码模板 controller-gen object paths=./...
3.controller-tools实战:自动生成代码
- 本项目已放入girhub代码仓库:https://github.com/graham924/share-code-operator-study
3.1.初始化项目
- 创建目录
- 初始化go项目,并get client-go
cd controller-tools-demo go mod init controller-tools-demo go get k8s.io/client-go go get k8s.io/apimachinery
3.2.使用type-scaffold工具生成types.go
- 需要注意:
- type-scaffold并不会生成文件,而是生成types.go的内容,打印到控制台,我们需要手动copy到types.go文件中去
- 不过使用kubebuilder的时候,会帮我们生成types.go文件的
- 执行
type-scaffold --kind=Application
,得到types.go的内容[root@master controller-tools-demo]# type-scaffold --kind=Application // ApplicationSpec defines the desired state of Application type ApplicationSpec struct { // INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster } // ApplicationStatus defines the observed state of Application. // It should always be reconstructable from the state of the cluster and/or outside world. type ApplicationStatus struct { // INSERT ADDITIONAL STATUS FIELDS -- observed state of cluster } // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // Application is the Schema for the applications API // +k8s:openapi-gen=true type Application struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec ApplicationSpec `json:"spec,omitempty"` Status ApplicationStatus `json:"status,omitempty"` } // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // ApplicationList contains a list of Application type ApplicationList struct { metav1.TypeMeta `json:",inline"` metav1.ListMeta `json:"metadata,omitempty"` Items []Application `json:"items"` }
- 在 v1alpha1 目录下创建 types.go 文件,将控制台的内容copy进去,记得导包
package v1alpha1 import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" // ApplicationSpec defines the desired state of Application type ApplicationSpec struct { // INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster } // ApplicationStatus defines the observed state of Application. // It should always be reconstructable from the state of the cluster and/or outside world. type ApplicationStatus struct { // INSERT ADDITIONAL STATUS FIELDS -- observed state of cluster } // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // Application is the Schema for the applications API // +k8s:openapi-gen=true type Application struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec ApplicationSpec `json:"spec,omitempty"` Status ApplicationStatus `json:"status,omitempty"` } // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // ApplicationList contains a list of Application type ApplicationList struct { metav1.TypeMeta `json:",inline"` metav1.ListMeta `json:"metadata,omitempty"` Items []Application `json:"items"` }
3.3.使用controller-gen生成deepcopy和crd文件
3.3.1.controller-gen --help 查看帮助文档
- 帮助文档给出了很多 examples
[root@master v1alpha1]# controller-gen --help Generate Kubernetes API extension resources and code. Usage: controller-gen [flags] Examples: # Generate RBAC manifests and crds for all types under apis/, # outputting crds to /tmp/crds and everything else to stdout controller-gen rbac:roleName=<role name> crd paths=./apis/... output:crd:dir=/tmp/crds output:stdout # Generate deepcopy/runtime.Object implementations for a particular file controller-gen object paths=./apis/v1beta1/some_types.go # Generate OpenAPI v3 schemas for API packages and merge them into existing CRD manifests controller-gen schemapatch:manifests=./manifests output:dir=./manifests paths=./pkg/apis/... # Run all the generators for a given project controller-gen paths=./apis/... # Explain the markers for generating CRDs, and their arguments controller-gen crd -ww Flags: -h, --detailed-help count print out more detailed help (up to -hhh for the most detailed output, or -hhhh for json output) --help print out usage and a summary of options --version show version -w, --which-markers count print out all markers available with the requested generators (up to -www for the most detailed output, or -wwww for json output) Options generators +webhook package generates (partial) {Mutating,Validating}WebhookConfiguration objects. +schemapatch[:generateEmbeddedObjectMeta=<bool>],manifests=<string>[,maxDescLen=<int>] package patches existing CRDs with new schemata. +rbac:roleName=<string> package generates ClusterRole objects. +object[:headerFile=<string>][,year=<string>] package generates code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations. +crd[:allowDangerousTypes=<bool>][,crdVersions=<[]string>][,generateEmbeddedObjectMeta=<bool>][,ignoreUnexportedFields=<bool>][,maxDescLen=<int>] package generates CustomResourceDefinition objects. generic +paths=<[]string> package represents paths and go-style path patterns to use as package roots. output rules (optionally as output:<generator>:...) +output:artifacts[:code=<string>],config=<string> package outputs artifacts to different locations, depending on whether they're package-associated or not. +output:dir=<string> package outputs each artifact to the given directory, regardless of if it's package-associated or not. +output:none package skips outputting anything. +output:stdout package outputs everything to standard-out, with no separation.
3.3.2.使用controller-gen生成deepcopy
- 先刷新一下包
go mod tidy
- 执行命令
cd controller-tools-demo controller-gen object paths=pkg/apis/appcontroller/v1alpha1/types.go
- 执行后,查看目录文件,发现生成了一个
zz_generated.deepcopy.go
文件[root@master controller-tools-demo]# tree . ├── go.mod ├── go.sum └── pkg └── apis └── appcontroller └── v1alpha1 ├── types.go └── zz_generated.deepcopy.go
- 查看
zz_generated.deepcopy.go
文件内容//go:build !ignore_autogenerated // +build !ignore_autogenerated // Code generated by controller-gen. DO NOT EDIT. package v1alpha1 import ( runtime "k8s.io/apimachinery/pkg/runtime" ) // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *Application) DeepCopyInto(out *Application) { *out = *in out.TypeMeta = in.TypeMeta in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) out.Spec = in.Spec out.Status = in.Status } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Application. func (in *Application) DeepCopy() *Application { if in == nil { return nil } out := new(Application) in.DeepCopyInto(out) return out } // DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. func (in *Application) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *ApplicationList) DeepCopyInto(out *ApplicationList) { *out = *in out.TypeMeta = in.TypeMeta in.ListMeta.DeepCopyInto(&out.ListMeta) if in.Items != nil { in, out := &in.Items, &out.Items *out = make([]Application, len(*in)) for i := range *in { (*in)[i].DeepCopyInto(&(*out)[i]) } } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApplicationList. func (in *ApplicationList) DeepCopy() *ApplicationList { if in == nil { return nil } out := new(ApplicationList) in.DeepCopyInto(out) return out } // DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. func (in *ApplicationList) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil }
3.3.3.使用controller-gen生成crd
- 先刷新一下包
go mod tidy
- 执行命令
cd controller-tools-demo controller-gen crd paths=./... output:crd:dir=config/crd
paths=./...
表示将当前目录下的所有子目录都包括在生成过程中output:crd:dir=config/crd
指定了输出目录为config/crd
- 执行后,生成目录
config
和 文件_.yaml
[root@master controller-tools-demo]# tree . ├── config │ └── crd │ └── _.yaml ├── go.mod ├── go.sum └── pkg └── apis └── appcontroller └── v1alpha1 ├── types.go └── zz_generated.deepcopy.go
- 文件
_.yaml
没有名字?指定groupName后重新生成 crd 文件- 因为我们没有给它指定 groupName,所以生成的yaml文件默认没有名字
- 我们在
pkg/apis/appcontroller/v1alpha1
下创建一个doc.go
文件,并在里面写上如下内容// +groupName=appcontroller.k8s.io package v1alpha1
- 然后删除原config后,重新生成crd
cd controller-tools-demo rm -rf config controller-gen crd paths=./... output:crd:dir=config/crd
- 生成完成后,目录如下
- crd文件名称为:appcontroller.k8s.io_applications.yaml
[root@master controller-tools-demo]# tree . ├── config │ └── crd │ └── appcontroller.k8s.io_applications.yaml ├── go.mod ├── go.sum └── pkg └── apis └── appcontroller └── v1alpha1 ├── doc.go ├── types.go └── zz_generated.deepcopy.go 6 directories, 6 files
- appcontroller.k8s.io_applications.yaml 内容如下
- 可以看到,
openAPIV3Schema.properties.spec
下面,没有 properties 项,因为我们的ApplicationSpec 为空,内部一个属性都没有。
--- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: (devel) creationTimestamp: null name: applications.appcontroller.k8s.io spec: group: appcontroller.k8s.io names: kind: Application listKind: ApplicationList plural: applications singular: application scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: Application is the Schema for the applications API properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: ApplicationSpec defines the desired state of Application type: object status: description: ApplicationStatus defines the observed state of Application. It should always be reconstructable from the state of the cluster and/or outside world. type: object type: object served: true storage: true
- 可以看到,
3.4.为ApplicationSpec添加内容,并重新生成crd文件
- 修改types.go,在 ApplicationSpec 中添加两个字段
type ApplicationSpec struct { // INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster Name string `json:"name"` Replicas int32 `json:"replicas"` }
- 删除原config后,重新生成crd文件
cd controller-tools-demo rm -rf config controller-gen crd paths=./... output:crd:dir=config/crd
- 新的crd文件内容如下:
- 可以看到,
openAPIV3Schema.properties.spec
下面,出现了 properties 项,因为我们为ApplicationSpec 添加了Name、Replicas两个值。
--- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: (devel) creationTimestamp: null name: applications.appcontroller.k8s.io spec: group: appcontroller.k8s.io names: kind: Application listKind: ApplicationList plural: applications singular: application scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: Application is the Schema for the applications API properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: ApplicationSpec defines the desired state of Application properties: name: description: INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster type: string replicas: format: int32 type: integer required: - name - replicas type: object status: description: ApplicationStatus defines the observed state of Application. It should always be reconstructable from the state of the cluster and/or outside world. type: object type: object served: true storage: true
- 可以看到,
3.5.手动注册版本v1alpha1的CRD资源
- 在生成了客户端代码后,我们还需要手动注册版本v1alpha1的CRD资源,才能真正使用这个client,不然在编译时会出现 undefined: v1alpha1.AddToScheme 错误、undefined: v1alpha1.Resource 错误。
- v1alpha1.AddToScheme、v1alpha1.Resource 这两个是用于 client 注册的
- 编写 v1alpha1/register.go
package v1alpha1 import ( "controller-tools-demo/pkg/apis/appcontroller" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" ) // SchemeGroupVersion is group version used to register these objects var SchemeGroupVersion = schema.GroupVersion{Group: appcontroller.GroupName, Version: "v1alpha1"} // Kind takes an unqualified kind and returns back a Group qualified GroupKind func Kind(kind string) schema.GroupKind { return SchemeGroupVersion.WithKind(kind).GroupKind() } // Resource takes an unqualified resource and returns a Group qualified GroupResource func Resource(resource string) schema.GroupResource { return SchemeGroupVersion.WithResource(resource).GroupResource() } var ( // SchemeBuilder initializes a scheme builder SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes) // AddToScheme is a global function that registers this API group & version to a scheme AddToScheme = SchemeBuilder.AddToScheme ) // Adds the list of known types to Scheme. func addKnownTypes(scheme *runtime.Scheme) error { scheme.AddKnownTypes(SchemeGroupVersion, &Application{}, &ApplicationList{}, ) metav1.AddToGroupVersion(scheme, SchemeGroupVersion) return nil }
3.6.在kubernetes集群中应用CRD
3.6.1.kubectl apply crd 资源
- 上面我们已经使用 controller-gen 自动生成了 CRD 文件,名称为
appcontroller.k8s.io_applications.yaml
,再 config/crd 目录下 - 我们需要使用
kubectl apply
命令创建CR资源cd controller-tools-demo kubectl apply -f config/crd/appcontroller.k8s.io_applications.yaml
- 有可能报错:
The CustomResourceDefinition "applications.appcontroller.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111
- 解决方法
- 这是kubernetes的保护机制,防止外部随意创建crd,破坏环境
- 报错中已经给了提示,查看github:https://github.com/kubernetes/enhancements/pull/1111
- 只需要在crd中,添加一条 annotation,然后再执行 kubectl apply -f 命令就可以了
metadata: annotations: api-approved.kubernetes.io: "https://github.com/kubernetes/kubernetes/pull/78458"
- 添加之后,完整的 crd 文件内容如下:
--- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: (devel) api-approved.kubernetes.io: "https://github.com/kubernetes/kubernetes/pull/78458" creationTimestamp: null name: applications.appcontroller.k8s.io spec: group: appcontroller.k8s.io names: kind: Application listKind: ApplicationList plural: applications singular: application scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: Application is the Schema for the applications API properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: ApplicationSpec defines the desired state of Application properties: name: description: INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster type: string replicas: format: int32 type: integer required: - name - replicas type: object status: description: ApplicationStatus defines the observed state of Application. It should always be reconstructable from the state of the cluster and/or outside world. type: object type: object served: true storage: true
3.6.2.编写 crd 资源的 example
- 在 config 目录下,创建一个example目录,内部可以编写一些 application资源的yaml,用于测试Application资源的创建是否可以成功
- 在config/example目录下,编写一个test_app.yaml
apiVersion: appcontroller.k8s.io/v1alpha1 kind: Application metadata: name: testapp namespace: tcs spec: name: testapp1 replicas: 2
- 创建Application资源
cd controller-tools-demo kubectl apply -f ./config/example/test_app.yaml
3.7.测试Application资源的获取
-
在项目根目录下,创建一个cmd目录,里面创建一个main.go文件
-
下面我们演示如何使用 code-generator 为 Application 的 v1alpha1 生成的客户端 AppcontrollerV1alpha1Client
-
编写main.go
package main import ( "context" "controller-tools-demo/pkg/apis/appcontroller/v1alpha1" "fmt" "log" "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/rest" "k8s.io/client-go/tools/clientcmd" ) func main() { config, err := clientcmd.BuildConfigFromFlags("", clientcmd.RecommendedHomeFile) if err != nil { log.Fatalln(err) } config.APIPath = "/apis/" config.GroupVersion = &v1alpha1.SchemeGroupVersion config.NegotiatedSerializer = scheme.Codecs client, err := rest.RESTClientFor(config) if err != nil { log.Fatalln(err) } app := v1alpha1.Application{} err = client.Get().Namespace("tcs").Resource("applications").Name("testapp").Do(context.TODO()).Into(&app) if err != nil { log.Fatalln(err) } newObj := app.DeepCopy() newObj.Spec.Name = "testapp2" fmt.Println(app.Spec.Name) fmt.Println(app.Spec.Replicas) fmt.Println(newObj.Spec.Name) }
-
输出结果
[root@master controller-tools-demo]# go run cmd/main.go testapp1 2 testapp2
-
如果没有在kubernetes集群中 应用 CRD资源,直接执行上面的代码,会报错:找不到这个资源。因此一定要先执行 3.6 的步骤
go run cmd/main.go 2024/01/31 16:01:17 the server could not find the requested resource (get applications.appcontroller.k8s.io test_app)