云计算虚拟化：k8s进阶-CRD项目部署

一. 前言

上一篇介绍了一个简单的，和MYSQL相关的CRD项目示例，并做了k8s集群本地运行测试，下面我们继续深入，把CRD项目部署到K8S集群内部，K8S进阶第一篇已经介绍了大致流程，本文主要涉及和部署相关的细节部分。

二. 部署步骤

2.1 make install
这部分前两篇都有涉及，不再累述。

2.2 check Dockerfile&yaml
这部分第一篇也有涉及，因为比较重要，这里再重述下，镜像部分可能会随时发生变更，请使用最新有效的镜像。
[root@k8s01 mydemo]# vi Dockerfile
# Build the manager binary
FROM golang:1.13 as builder

WORKDIR /workspace
ENV GOPROXY https://goproxy.cn/
ENV GO111MODULE on
…
#FROM gcr.io/distroless/static:nonroot
FROM registry.cn-hangzhou.aliyuncs.com/byteforce/distroless:nonroot
…
[root@k8s01 mydemo]# vi config/default/manager_auth_proxy_patch.yaml
…
#image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
image: registry.cn-hangzhou.aliyuncs.com/hsc/kube-rbac-proxy
image: registry.cn-hangzhou.aliyuncs.com/kubenode/kube-rbac-proxy:v0.4.0
…
2.3 部署
2.3.1 创建docker image
root@k8s01 K8S-CRD-Demo]# make docker-build docker-push IMG=registry.ips.com.cn/demo/mycontroller:v0.1.0
which: no controller-gen in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin:/usr/local/kubebuilder/bin)
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/root/go/bin/controller-gen object:headerFile=“hack/boilerplate.go.txt” paths="./…"
go fmt ./…
go vet ./…
go: downloading github.com/onsi/gomega v1.8.1
go: downloading github.com/onsi/ginkgo v1.11.0
go: downloading golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7
go: downloading github.com/hpcloud/tail v1.0.0
go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
/root/go/bin/controller-gen “crd:trivialVersions=true” rbac:roleName=manager-role webhook paths="./…" output:crd:artifacts:config=config/crd/bases
go test ./… -coverprofile cover.out
? K8S-CRD-Demo [no test files]
? K8S-CRD-Demo/api/v1 [no test files]
ok K8S-CRD-Demo/controllers 6.123s coverage: 0.0% of statements
docker build . -t registry.ips.com.cn/demo/mycontroller:v0.1.0
Sending build context to Docker daemon 40.35MB
Step 1/16 : FROM golang:1.13 as builder
1.13: Pulling from library/golang
d6ff36c9ec48: Downloading [===========> ] 11.12MB/50.4MB
c958d65b3090: Downloading
edaf0a6b092f: Download complete
80931cf68816: Downloading [===========> ] 10.89MB/51.83MB
813643441356: Downloading
799f41bb59c9: Waiting
16b5038bccc8: Waiting
1.13: Pulling from library/golang
d6ff36c9ec48: Pull complete
c958d65b3090: Pull complete
edaf0a6b092f: Pull complete
80931cf68816: Pull complete
813643441356: Pull complete
799f41bb59c9: Pull complete
16b5038bccc8: Pull complete
Digest: sha256:8ebb6d5a48deef738381b56b1d4cd33d99a5d608e0d03c5fe8dfa3f68d41a1f8
Status: Downloaded newer image for golang:1.13
—> d6f3656320fe
Step 2/16 : WORKDIR /workspace
—> Running in d1d69ffab6e8
Removing intermediate container d1d69ffab6e8
—> 3bf6b845f6eb
Step 3/16 : ENV GOPROXY https://goproxy.cn/
—> Running in cd216a5720cc
Removing intermediate container cd216a5720cc
—> 440ba9c74f05
Step 4/16 : ENV GO111MODULE on
—> Running in 653b6492cc5d
Removing intermediate container 653b6492cc5d
—> 0531f4d84e92
Step 5/16 : COPY go.mod go.mod
—> 8bac793c1061
Step 6/16 : COPY go.sum go.sum
—> 462a2041fc12
Step 7/16 : RUN go mod download
—> Running in eef58d5d7d3f
go: finding cloud.google.com/go v0.38.0
go: finding github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78
go: finding github.com/Azure/go-autorest/autorest v0.9.0
go: finding github.com/Azure/go-autorest/autorest/adal v0.5.0
go: finding github.com/Azure/go-autorest/autorest/date v0.1.0
go: finding github.com/Azure/go-autorest/autorest/mocks v0.2.0
go: finding github.com/Azure/go-autorest/logger v0.1.0
go: finding github.com/Azure/go-autorest/tracing v0.5.0
go: finding github.com/BurntSushi/toml v0.3.1
go: finding github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802
go: finding github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46
go: finding github.com/PuerkitoBio/purell v1.1.1
go: finding github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: finding github.com/agnivade/levenshtein v1.0.1
go: finding github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc
go: finding github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: finding github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883
go: finding github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6
go: finding github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: finding github.com/beorn7/perks v1.0.0
go: finding github.com/bgentry/speakeasy v0.1.0
go: finding github.com/blang/semver v3.5.0+incompatible
go: finding github.com/client9/misspell v0.3.4
go: finding github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa
go: finding github.com/coreos/etcd v3.3.10+incompatible
go: finding github.com/coreos/go-etcd v2.0.0+incompatible
go: finding github.com/coreos/go-oidc v2.1.0+incompatible
go: finding github.com/coreos/go-semver v0.3.0
go: finding github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e
go: finding github.com/coreos/pkg v0.0.0-20180108230652-97fdf19511ea
go: finding github.com/cpuguy83/go-md2man v1.0.10
go: finding github.com/creack/pty v1.1.7
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: finding github.com/docker/docker v0.7.3-0.20190327010347-be7ac8be2ae0
go: finding github.com/docker/go-units v0.4.0
go: finding github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96
go: finding github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815
go: finding github.com/dustin/go-humanize v1.0.0
go: finding github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e
go: finding github.com/emicklei/go-restful v2.9.5+incompatible
go: finding github.com/evanphx/json-patch v4.5.0+incompatible
go: finding github.com/fatih/color v1.7.0
go: finding github.com/fsnotify/fsnotify v1.4.7
go: finding github.com/ghodss/yaml v1.0.0
go: finding github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
go: finding github.com/go-kit/kit v0.8.0
go: finding github.com/go-logfmt/logfmt v0.3.0
go: finding github.com/go-logr/logr v0.1.0
go: finding github.com/go-logr/zapr v0.1.0
go: finding github.com/go-openapi/analysis v0.19.5
go: finding github.com/go-openapi/errors v0.19.2
go: finding github.com/go-openapi/jsonpointer v0.19.3
go: finding github.com/go-openapi/jsonreference v0.19.3
go: finding github.com/go-openapi/loads v0.19.4
go: finding github.com/go-openapi/runtime v0.19.4
go: finding github.com/go-openapi/spec v0.19.3
go: finding github.com/go-openapi/strfmt v0.19.3
go: finding github.com/go-openapi/swag v0.19.5
go: finding github.com/go-openapi/validate v0.19.5
go: finding github.com/go-stack/stack v1.8.0
go: finding github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d
go: finding github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: finding github.com/golang/groupcache v0.0.0-20180513044358-24b0969c4cb7
go: finding github.com/golang/mock v1.2.0
go: finding github.com/golang/protobuf v1.3.2
go: finding github.com/google/btree v1.0.0
go: finding github.com/google/go-cmp v0.3.0
go: finding github.com/google/gofuzz v1.0.0
go: finding github.com/google/martian v2.1.0+incompatible
go: finding github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57
go: finding github.com/google/uuid v1.1.1
go: finding github.com/googleapis/gax-go/v2 v2.0.4
go: finding github.com/googleapis/gnostic v0.3.1
go: finding github.com/gophercloud/gophercloud v0.1.0
go: finding github.com/gorilla/websocket v1.4.0
go: finding github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go: finding github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
go: finding github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: finding github.com/grpc-ecosystem/grpc-gateway v1.9.5
go: finding github.com/hashicorp/golang-lru v0.5.1
go: finding github.com/hashicorp/hcl v1.0.0
go: finding github.com/hpcloud/tail v1.0.0
go: finding github.com/imdario/mergo v0.3.6
go: finding github.com/inconshreveable/mousetrap v1.0.0
go: finding github.com/jonboulle/clockwork v0.1.0
go: finding github.com/json-iterator/go v1.1.8
go: finding github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024
go: finding github.com/julienschmidt/httprouter v1.2.0
go: finding github.com/kisielk/errcheck v1.2.0
go: finding github.com/kisielk/gotool v1.0.0
go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.1
go: finding github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
go: finding github.com/kr/pretty v0.1.0
go: finding github.com/kr/pty v1.1.5
go: finding github.com/kr/text v0.1.0
go: finding github.com/magiconair/properties v1.8.0
go: finding github.com/mailru/easyjson v0.7.0
go: finding github.com/mattn/go-colorable v0.0.9
go: finding github.com/mattn/go-isatty v0.0.4
go: finding github.com/mattn/go-runewidth v0.0.2
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/mitchellh/go-homedir v1.1.0
go: finding github.com/mitchellh/mapstructure v1.1.2
go: finding github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: finding github.com/modern-go/reflect2 v1.0.1
go: finding github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822
go: finding github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223
go: finding github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f
go: finding github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
go: finding github.com/onsi/ginkgo v1.11.0
go: finding github.com/onsi/gomega v1.8.1
go: finding github.com/pborman/uuid v1.2.0
go: finding github.com/pelletier/go-toml v1.2.0
go: finding github.com/peterbourgon/diskv v2.0.1+incompatible
go: finding github.com/pkg/errors v0.8.1
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021
go: finding github.com/prometheus/client_golang v1.0.0
go: finding github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90
go: finding github.com/prometheus/common v0.4.1
go: finding github.com/prometheus/procfs v0.0.2
go: finding github.com/remyoudompheng/bigfft v0.0.0-20170806203942-52369c62f446
go: finding github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af
go: finding github.com/russross/blackfriday v1.5.2
go: finding github.com/sergi/go-diff v1.0.0
go: finding github.com/sirupsen/logrus v1.4.2
go: finding github.com/soheilhy/cmux v0.1.4
go: finding github.com/spf13/afero v1.2.2
go: finding github.com/spf13/cast v1.3.0
go: finding github.com/spf13/cobra v0.0.5
go: finding github.com/spf13/jwalterweatherman v1.0.0
go: finding github.com/spf13/pflag v1.0.5
go: finding github.com/spf13/viper v1.3.2
go: finding github.com/stretchr/objx v0.2.0
go: finding github.com/stretchr/testify v1.4.0
go: finding github.com/tidwall/pretty v1.0.0
go: finding github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
go: finding github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8
go: finding github.com/urfave/cli v1.20.0
go: finding github.com/vektah/gqlparser v1.1.2
go: finding github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: finding github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77
go: finding go.etcd.io/bbolt v1.3.3
go: finding go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738
go: finding go.mongodb.org/mongo-driver v1.1.2
go: finding go.opencensus.io v0.21.0
go: finding go.uber.org/atomic v1.3.2
go: finding go.uber.org/multierr v1.1.0
go: finding go.uber.org/zap v1.10.0
go: finding golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586
go: finding golang.org/x/exp v0.0.0-20190312203227-4b39c73a6495
go: finding golang.org/x/image v0.0.0-20190227222117-0694c2d4d067
go: finding golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3
go: finding golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6
go: finding golang.org/x/net v0.0.0-20191004110552-13f9640d40b9
go: finding golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go: finding golang.org/x/sync v0.0.0-20190423024810-112230192c58
go: finding golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456
go: finding golang.org/x/text v0.3.2
go: finding golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go: finding golang.org/x/tools v0.0.0-20190920225731-5eefd052ad72
go: finding golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7
go: finding gomodules.xyz/jsonpatch/v2 v2.0.1
go: finding gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485
go: finding gonum.org/v1/netlib v0.0.0-20190331212654-76723241ea4e
go: finding google.golang.org/api v0.4.0
go: finding google.golang.org/appengine v1.5.0
go: finding google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873
go: finding google.golang.org/grpc v1.23.1
go: finding gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: finding gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: finding gopkg.in/cheggaaa/pb.v1 v1.0.25
go: finding gopkg.in/fsnotify.v1 v1.4.7
go: finding gopkg.in/inf.v0 v0.9.1
go: finding gopkg.in/natefinch/lumberjack.v2 v2.0.0
go: finding gopkg.in/resty.v1 v1.12.0
go: finding gopkg.in/square/go-jose.v2 v2.2.2
go: finding gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: finding gopkg.in/yaml.v2 v2.2.4
go: finding gotest.tools v2.2.0+incompatible
go: finding honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc
go: finding k8s.io/api v0.17.2
go: finding k8s.io/apiextensions-apiserver v0.17.2
go: finding k8s.io/apimachinery v0.17.2
go: finding k8s.io/apiserver v0.17.2
go: finding k8s.io/client-go v0.17.2
go: finding k8s.io/code-generator v0.17.2
go: finding k8s.io/component-base v0.17.2
go: finding k8s.io/gengo v0.0.0-20190822140433-26a664648505
go: finding k8s.io/klog v1.0.0
go: finding k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a
go: finding k8s.io/utils v0.0.0-20191114184206-e782cd3c129f
go: finding modernc.org/cc v1.0.0
go: finding modernc.org/golex v1.0.0
go: finding modernc.org/mathutil v1.0.0
go: finding modernc.org/strutil v1.0.0
go: finding modernc.org/xc v1.0.0
go: finding sigs.k8s.io/controller-runtime v0.5.0
go: finding sigs.k8s.io/structured-merge-diff v1.0.1-0.20191108220359-b1b620dd3f06
go: finding sigs.k8s.io/yaml v1.1.0
Removing intermediate container eef58d5d7d3f
—> 7c447875326e
Step 8/16 : COPY main.go main.go
—> 53283e3d732a
Step 9/16 : COPY api/ api/
—> 895005f49145
Step 10/16 : COPY controllers/ controllers/
—> fb0d4b5270cb
Step 11/16 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go
—> Running in 6f74881f1d7e
Removing intermediate container 6f74881f1d7e
—> 2aed192f48a7
Step 12/16 : FROM registry.cn-hangzhou.aliyuncs.com/byteforce/distroless:nonroot
nonroot: Pulling from byteforce/distroless
4000adbbc3eb: Pull complete
3c2cba919283: Pull complete
Digest: sha256:d0f414c64bdb0ceebea651b14ad5be9a281a7a1d67751f9de419316390239b62
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/byteforce/distroless:nonroot
—> 25b8fd42ff36
Step 13/16 : WORKDIR /
—> Running in 4e9ec37c3b8e
Removing intermediate container 4e9ec37c3b8e
—> 5f9f617e4efa
Step 14/16 : COPY --from=builder /workspace/manager .
—> dc95d0f548cf
Step 15/16 : USER nonroot:nonroot
—> Running in 0c6439a5121a
Removing intermediate container 0c6439a5121a
—> e88a2bfd301a
Step 16/16 : ENTRYPOINT ["/manager"]
—> Running in 9181c4f94fc3
Removing intermediate container 9181c4f94fc3
—> 39220582249a
Successfully built 39220582249a
Successfully tagged registry.ips.com.cn/demo/mycontroller:v0.1.0
docker push registry.ips.com.cn/demo/mycontroller:v0.1.0
The push refers to repository [registry.ips.com.cn/demo/mycontroller]
Get https://registry.ips.com.cn/v2/: dial tcp: lookup registry.ips.com.cn on 114.114.114.114:53: no such host
make: *** [docker-push] Error 1
解决方法：如果无法连接镜像仓库，可以导出导入镜像到对应节点

2.3.2 导出导入镜像；如果镜像仓库可用，请跳过此步。
[root@k8s01 K8S-CRD-Demo]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.ips.com.cn/demo/mycontroller v0.1.0 0a62c7b3c0cc 9 seconds ago 43.9MB
…
[root@k8s01 home]# docker save -o mycon.tar registry.ips.com.cn/demo/mycontroller:v0.1.0
[root@k8s01 home]# scp mycon.tar k8s02:/home/
[root@k8s01 home]# scp mycon.tar k8s03:/home/
[root@k8s02 home]# docker load -i mycon.tar
d72a74c56330: Loading layer [==================================================>] 3.031MB/3.031MB
84ff92691f90: Loading layer [==================================================>] 10.24kB/10.24kB
8a1827160e35: Loading layer [==================================================>] 42.12MB/42.12MB
Loaded image: registry.ips.com.cn/demo/mycontroller:v0.1.0
[root@k8s02 home]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.ips.com.cn/demo/mycontroller v0.1.0 0a62c7b3c0cc 2 minutes ago 43.9MB
mysql 5.7 f07dfa83b528 34 hours ago 448MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 23 months ago 52.6MB
cargo.caicloud.io/caicloud/pause-amd64 3.1 da86e6ba6ca1 3 years ago 742kB
[root@k8s03 home]# docker load -i mycon.tar
d72a74c56330: Loading layer [==================================================>] 3.031MB/3.031MB
84ff92691f90: Loading layer [==================================================>] 10.24kB/10.24kB
8a1827160e35: Loading layer [==================================================>] 42.12MB/42.12MB
Loaded image: registry.ips.com.cn/demo/mycontroller:v0.1.0
[root@k8s03 home]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.ips.com.cn/demo/mycontroller v0.1.0 0a62c7b3c0cc About a minute ago 43.9MB
mysql 5.7 697daaecf703 11 days ago 448MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 23 months ago 52.6MB
cargo.caicloud.io/caicloud/pause-amd64 3.1 da86e6ba6ca1 3 years ago 742kB
[root@k8s03 home]#

2.3.3 部署
[root@k8s01 K8S-CRD-Demo]# make deploy IMG=registry.ips.com.cn/demo/mycontroller:v0.1.0
which: no controller-gen in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin:/usr/local/kubebuilder/bin)
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/root/go/bin/controller-gen “crd:trivialVersions=true” rbac:roleName=manager-role webhook paths="./…" output:crd:artifacts:config=config/crd/bases
cd config/manager && kustomize edit set image controller=registry.ips.com.cn/demo/mycontroller:v0.1.0
kustomize build config/default | kubectl apply -f -
namespace/crd-demo-system created
customresourcedefinition.apiextensions.k8s.io/mykinds.mygroup.ips.com.cn configured
role.rbac.authorization.k8s.io/crd-demo-leader-election-role created
clusterrole.rbac.authorization.k8s.io/crd-demo-manager-role created
clusterrole.rbac.authorization.k8s.io/crd-demo-proxy-role created
clusterrole.rbac.authorization.k8s.io/crd-demo-metrics-reader created
rolebinding.rbac.authorization.k8s.io/crd-demo-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/crd-demo-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/crd-demo-proxy-rolebinding created
service/crd-demo-controller-manager-metrics-service created
deployment.apps/crd-demo-controller-manager created
[root@k8s01 K8S-CRD-Demo]# kubectl get ns
NAME STATUS AGE
crd-demo-system Active 6m23s
default Active 69m
kube-node-lease Active 70m
kube-public Active 70m
kube-system Active 70m
[root@k8s01 K8S-CRD-Demo]# kubectl get crd
NAME CREATED AT
mykinds.mygroup.ips.com.cn 2020-11-24T06:11:32Z
[root@k8s01 K8S-CRD-Demo]# kubectl get deploy -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
crd-demo-system crd-demo-controller-manager 1/1 1 1 4m43s
[root@k8s01 K8S-CRD-Demo]# kubectl get rs -A
NAMESPACE NAME DESIRED CURRENT READY AGE
crd-demo-system crd-demo-controller-manager-6bf744659 1 1 1 4m51s
[root@k8s01 K8S-CRD-Demo]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
crd-demo-system crd-demo-controller-manager-6bf744659-ljh5s 2/2 Running 0 4m58s
kube-system kube-flannel-ds-amd64-gqn2l 1/1 Running 0 67m
kube-system kube-flannel-ds-amd64-zcd9c 1/1 Running 0 67m
[root@k8s01 K8S-CRD-Demo]# kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
crd-demo-system crd-demo-controller-manager-metrics-service ClusterIP 196.169.45.113 8443/TCP 5m3s
default kubernetes ClusterIP 196.169.0.1 443/TCP 68m

三. 测试

[root@k8s01 samples]# vi mycrd_mysql.yaml

apiVersion: mygroup.ips.com.cn/v1  # 这就是上一篇博文中我们自己定义的apiserver版本
kind: Mykind                       # 这就是上一篇博文中我们自己定义的crd kind
metadata:                       
  name: mykind-sample              # 这里我们自己定义mykind的名称，全局唯一
spec:                            #Spec部分就是我们整合deploy&service关键要素后需要自定义的元素
  replicas: 1                    # <integer> Pod副本期待数量
  image: mysql:5.7               # <string> image版本   
  port: 3306                     # <integer> mysql port
  nodeport: 30306                # <integer> mysql port for outside
  env:                           # <object> 写入到容器内的环境变量，此格式主要为了复用api中已有数据结构,,也可以分开定义
    - name: MYSQL_ROOT_PASSWORD  # <string> mysql的root密码的变量
      value: "123456"            # <string> 密码内容

[root@k8s01 samples]# kubectl apply -f mycrd_mysql.yaml
mykind.mygroup.ips.com.cn/mykind-sample created
[root@k8s01 samples]# kubectl get mykind
NAME AGE
mykind-sample 4m51s
[root@k8s01 samples]# kubectl logs crd-demo-controller-manager-6bf744659-ljh5s -n crd-demo-system manager
2020-11-24T07:01:02.414Z INFO controller-runtime.metrics metrics server is starting to listen {“addr”: “127.0.0.1:8080”}
2020-11-24T07:01:02.414Z INFO setup starting manager
I1231 07:01:02.414599 1 leaderelection.go:242] attempting to acquire leader lease crd-demo-system/5bf6bb78.ips.com.cn…
2020-11-24T07:01:02.414Z INFO controller-runtime.manager starting metrics server {“path”: “/metrics”}
I1231 07:01:02.423292 1 leaderelection.go:252] successfully acquired lease crd-demo-system/5bf6bb78.ips.com.cn
2020-11-24T07:01:02.423Z DEBUG controller-runtime.manager.events Normal {“object”: {“kind”:“ConfigMap”,“namespace”:“crd-demo-system”,“name”:“5bf6bb78.ips.com.cn”,“uid”:“3218f400-b7fd-49ce-ad98-57de404be1bd”,“apiVersion”:“v1”,“resourceVersion”:“9134”}, “reason”: “LeaderElection”, “message”: “crd-demo-controller-manager-6bf744659-ljh5s_0aa34491-4817-49b7-8f22-f83050ecb2a5 became leader”}
2020-11-24T07:01:02.508Z INFO controller-runtime.controller Starting EventSource {“controller”: “mykind”, “source”: “kind source: /, Kind=”}
2020-11-24T07:01:02.610Z INFO controller-runtime.controller Starting Controller {“controller”: “mykind”}
2020-11-24T07:01:02.610Z INFO controller-runtime.controller Starting workers {“controller”: “mykind”, “worker count”: 1}
—start Reconcile—
E1231 07:16:12.071373 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Deployment: deployments.apps is forbidden: User “system:serviceaccount:crd-demo-system:default” cannot list resource “deployments” in API group “apps” at the cluster scope
E1231 07:16:13.072791 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Deployment: deployments.apps is forbidden: User “system:serviceaccount:crd-demo-system:default” cannot list resource “deployments” in API group “apps” at the cluster scope

解决方法：
========
[root@k8s01 samples]# vi bind.yaml

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:  
  name: crd-demo-default-binding
subjects:  
  - kind: ServiceAccount    
    name: default    
    namespace: crd-demo-system
roleRef:  
  kind: ClusterRole  
  name: cluster-admin  
  apiGroup: rbac.authorization.k8s.io

[root@k8s01 samples]# kubectl apply -f bind.yaml
clusterrolebinding.rbac.authorization.k8s.io/crd-demo-default-binding created
重新创建
[root@k8s01 samples]# kubectl apply -f mycrd_mysql.yaml
mykind.mygroup.ips.com.cn/mykind-sample created
[root@k8s01 samples]# kubectl get pods -A -owide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
crd-demo-system crd-demo-controller-manager-6bf744659-ljh5s 2/2 Running 0 46m 196.159.1.2 k8s03 <none> <none>
default mykind-sample-7584496b56-ljfb7 1/1 Running 0 6m58s 196.159.0.2 k8s02 <none> <none>
kube-system kube-flannel-ds-amd64-gqn2l 1/1 Running 0 109m 192.168.100.102 k8s02 <none> <none>
kube-system kube-flannel-ds-amd64-zcd9c 1/1 Running 0 109m 192.168.100.103 k8s03 <none> <none>
[root@k8s01 samples]# kubectl get svc -owide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 196.169.0.1 443/TCP 110m <none>
mykind-sample NodePort 196.169.37.225 <none> 3306:30306/TCP 7m12s app=mykind-sample

[root@k8s01 home]# ./mysql -hk8s02 -P30306 -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.32 MySQL Community Server (GPL)

Copyright © 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql> show databases;
±-------------------+
| Database |
±-------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
±-------------------+
4 rows in set (0.00 sec)

mysql>

四.可能遇到的问题

4.1 ImagePullBackOff
症状：
=====
[root@k8s01 K8S-CRD-Demo]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
crd-demo-system crd-demo-controller-manager-59cf997ccd-zmskk 1/2 ImagePullBackOff 2 105s
[root@k8s01 K8S-CRD-Demo]# kubectl describe pod crd-demo-controller-manager-59cf997ccd-zmskk -n crd-demo-system
…
Events:
Type Reason Age From Message

---

Normal Scheduled default-scheduler Successfully assigned crd-demo-system/crd-demo-controller-manager-59cf997ccd-zmskk to k8s02
Normal Pulled 102s (x2 over 2m13s) kubelet, k8s02 Container image “registry.ips.com.cn/demo/mycontroller:v0.1.0” already present on machine
Normal Created 102s (x2 over 2m13s) kubelet, k8s02 Created container manager
Normal Started 102s (x2 over 2m13s) kubelet, k8s02 Started container manager
Normal BackOff 101s (x5 over 2m13s) kubelet, k8s02 Back-off pulling image “registry.cn-hangzhou.aliyuncs.com/hsc/kube-rbac-proxy”
Warning Failed 101s (x5 over 2m13s) kubelet, k8s02 Error: ImagePullBackOff
Warning Failed 86s (x3 over 2m13s) kubelet, k8s02 Error: ErrImagePull
Normal Pulling 86s (x3 over 2m14s) kubelet, k8s02 Pulling image “registry.cn-hangzhou.aliyuncs.com/hsc/kube-rbac-proxy”
Warning Failed 86s (x3 over 2m13s) kubelet, k8s02 Failed to pull image “registry.cn-hangzhou.aliyuncs.com/hsc/kube-rbac-proxy”: rpc error: code = Unknown desc = Error response from daemon: pull access denied for registry.cn-hangzhou.aliyuncs.com/hsc/kube-rbac-proxy, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

解决方法：
========
[root@k8s01 mydemo]# vi config/default/manager_auth_proxy_patch.yaml
…
#image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
#image: registry.cn-hangzhou.aliyuncs.com/hsc/kube-rbac-proxy #invalid
image: registry.cn-hangzhou.aliyuncs.com/kubenode/kube-rbac-proxy:v0.4.0
然后删除系统创建的资源，从2.3.1创建Docker image重做。

4.2 Error from server (Forbidden): Forbidden
症状：
=====
[root@k8s01 samples]# kubectl logs crd-demo-controller-manager-6bf744659-vws9r -n crd-demo-system -c manager
Error from server (Forbidden): Forbidden (user=system:node:k8s02, verb=get, resource=nodes, subresource=proxy) ( pods/log crd-demo-controller-manager-6bf744659-vws9r)

解决方法：
========

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
EOF

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: system:nodes
EOF

4.3 timeout error
症状：
=====
[root@k8s02 containers]# kubectl logs crd-demo-controller-manager-6bf744659-77qhj -n crd-demo-system -c manager
2020-11-24T09:25:21.400Z ERROR controller-runtime.manager Failed to get API Group-Resources {“error”: “Get https://196.169.0.1:443/api?timeout=32s: dial tcp 196.169.0.1:443: i/o timeout”}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/manager.New
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/manager/manager.go:241
main.main
/workspace/main.go:57
runtime.main
/usr/local/go/src/runtime/proc.go:203
2020-11-24T09:25:21.401Z ERROR setup unable to start manager {“error”: “Get https://196.169.0.1:443/api?timeout=32s: dial tcp 196.169.0.1:443: i/o timeout”}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
main.main
/workspace/main.go:65
runtime.main
/usr/local/go/src/runtime/proc.go:203

解决方法：
========
调整kube-proxy参数：
vi /etc/kubernetes/kube-proxy-config.yaml

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/etc/kubernetes/kube-proxy.config"
mode: "iptables"
iptables:
  masqueradeAll: true

重启kubelet,kube-proxy