本文着重介绍本人在测试webhook过程中遇到的问题,及最终解决方案。
我在linux机器上测试,安装kubebuilder,kustomize等问题不再赘述。
一,安装项目
1.首先在gopath下新建目录,并进入目录
mkdir mywebhook
cd mywebhook/
2.初始化项目
[root@host mywebhook]# kubebuilder init --domain zander.com
3,生成kind,crd
kubebuilder create api --group webapp --version v1 --kind Guestbook
4.安装crd
make install
5 生成cr
kubectl apply -f config/samples/
6.生成webhook
kubebuilder create webhook --group webapp --version v1 --kind Guestbook --programmatic-validation --defaulting
一切到这里都还非常顺利,下面将进入一步一个坑的阶段
二,启动服务
1.证书问题
make run
报错,找不到k8s_webhook-server/serving-certs/tls.crt
解决办法
cp /etc/kubernetes/pki/apiserver.crt /tmp/k8s_webhook-server/serving-certs/tls.crt
cp /etc/kubernetes/pki/apiserver.key /tmp/k8s_webhook-server/serving-certs/tls.key
如果本地没有这个证书,也可以使用工具生成
然后make run可以正常启动了
2.webhook掉不到
这是由于apiserver根本就不知道有webhook
解决办法,修改config/default/kustomization.yaml
文件
将webhook相关配置打开,如下
# Adds namespace to all resources.
namespace: mywebhook-system
# Value of this field is prepended to the
# names of all resources, e.g. a deployment named
# "wordpress" becomes "alices-wordpress".
# Note that it should also match with the prefix (text before '-') of the namespace
# field above.
namePrefix: mywebhook-
# Labels to add to all resources and selectors.
#commonLabels:
# someName: someValue
bases:
- ../crd
- ../rbac
- ../manager
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
# crd/kustomization.yaml
- ../webhook
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
- ../certmanager
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
#- ../prometheus
patchesStrategicMerge:
# Protect the /metrics endpoint by putting it behind auth.
# If you want your controller-manager to expose the /metrics
# endpoint w/o any authn/z, please comment the following line.
- manager_auth_proxy_patch.yaml
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
# crd/kustomization.yaml
- manager_webhook_patch.yaml
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
# 'CERTMANAGER' needs to be enabled to use ca injection
- webhookcainjection_patch.yaml
# the following config is for teaching kustomize how to do var substitution
vars:
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
objref:
kind: Certificate
group: cert-manager.io
version: v1alpha2
name: serving-cert # this name should match the one in certificate.yaml
fieldref:
fieldpath: metadata.namespace
- name: CERTIFICATE_NAME
objref:
kind: Certificate
group: cert-manager.io
version: v1alpha2
name: serving-cert # this name should match the one in certificate.yaml
- name: SERVICE_NAMESPACE # namespace of the service
objref:
kind: Service
version: v1
name: webhook-service
fieldref:
fieldpath: metadata.namespace
- name: SERVICE_NAME
objref:
kind: Service
version: v1
name: webhook-service
3.生成controller镜像
make docker-build
报错go: github.com/go-logr/logr@v0.1.0: Get https://proxy.golang.org/github.com/go-logr/logr/@v/v0.1.0.mod: dial tcp 216.58.200.49:443: connect: connection refused
解决办法,修改镜像
将Dockerfile修改如下:
FROM registry.xxx/baseimages/centos:centos7.5.1804
WORKDIR /
COPY ./bin/manager .
ENTRYPOINT ["/manager"]
4.部署controller等
make deploy
查看容器状态
kubectl get pod -A
发现容器挂掉,镜像拉取失败
mywebhook-controller-manager-67f555b9bf-qf455 0/2
ErrImagePull
查看细节
kubectl logs -f mywebhook-controller-manager-67f555b9bf-qf455 -n mywebhook-system manager
发现报错
Error from server (BadRequest): container “manager” in pod “mywebhook-controller-manager-67f555b9bf-qf455” is waiting to start: trying and failing to pull image
说明是镜像拉取失败,由于我使用本地镜像,并没有docker-push到远程仓库,所以拉取失败
解决办法
将docker 镜像版本号修改,不使用latest ,以避免从远程拉取
修改makefile
IMG ?= guestbook-controller:1.0.0
manager: generate fmt vet manifests
go build -o bin/manager main.go
deploy: manager
cd config/manager && kustomize edit set image controller=${IMG}
kustomize build config/default | kubectl apply -f -
修改config/manager/manager.yaml
image: guestbook-controller:1.0.0
删除原先部署的内容,再执行make docker-build,make deploy
发现容器起来了
mywebhook-system mywebhook-controller-manager-698cc75d79-9t2zh 1/1 Running 0 21s
查看下日志
[root@host mywebhook]# kubectl logs -f mywebhook-controller-manager-698cc75d79-9t2zh -n mywebhook-system manager
2021-03-11T09:47:06.630Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2021-03-11T09:47:06.646Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "webapp.zander.com/v1, Kind=Guestbook", "path": "/mutate-webapp-zander-com-v1-guestbook"}
2021-03-11T09:47:06.646Z INFO controller-runtime.webhook registering webhook {"path": "/mutate-webapp-zander-com-v1-guestbook"}
2021-03-11T09:47:06.646Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "webapp.zander.com/v1, Kind=Guestbook", "path": "/validate-webapp-zander-com-v1-guestbook"}
2021-03-11T09:47:06.646Z INFO controller-runtime.webhook registering webhook {"path": "/validate-webapp-zander-com-v1-guestbook"}
2021-03-11T09:47:06.646Z INFO setup starting manager
2021-03-11T09:47:06.733Z INFO controller-runtime.webhook.webhooks starting webhook server
I0311 09:47:06.762562 1 leaderelection.go:242] attempting to acquire leader lease mywebhook-system/7117d994.zander.com...
2021-03-11T09:47:06.794Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2021-03-11T09:47:08.287Z INFO controller-runtime.certwatcher Updated current TLS certificate
2021-03-11T09:47:08.288Z INFO controller-runtime.webhook serving webhook server {"host": "", "port": 9443}
2021-03-11T09:47:08.307Z INFO controller-runtime.certwatcher Starting certificate watcher
I0311 09:47:12.954688 1 leaderelection.go:252] successfully acquired lease mywebhook-system/7117d994.zander.com
2021-03-11T09:47:13.371Z INFO controller-runtime.controller Starting EventSource {"controller": "guestbook", "source": "kind source: /, Kind="}
2021-03-11T09:47:13.273Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"mywebhook-system","name":"7117d994.zander.com","uid":"e25380ed-3bb6-422f-9231-b2182538e84e","apiVersion":"v1","resourceVersion":"18235607"}, "reason": "LeaderElection", "message": "mywebhook-controller-manager-698cc75d79-9t2zh_a5599b20-7d36-4adf-8f4b-21124531e831 became leader"}
2021-03-11T09:47:18.740Z INFO controller-runtime.controller Starting Controller {"controller": "guestbook"}
2021-03-11T09:47:18.813Z INFO controller-runtime.controller Starting workers {"controller": "guestbook", "worker count": 1}
可以看到webhook服务已经启动了
修改webapp_v1_guestbook.yaml
中参数,重新应用
kubectl apply -f config/samples/webapp_v1_guestbook.yaml
再看controller日志,发现webhook的日志打印出来了!
NFO guestbook-resource default {“name”: “guestbook-sample”}
2021-03-11T09:50:28.617Z DEBUG controller-runtime.webhook.webhooks wrote response {“webhook”: “/mutate-webapp-zander-com-v1-guestbook”, “UID”: “24beb268-c42c-42c9-989c-1901882b7b6f”, “allowed”: true, “result”: {}, “resultError”: “got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}”}
2021-03-11T09:50:29.045Z DEBUG controller-runtime.webhook.webhooks received request {“webhook”: “/validate-webapp-zander-com-v1-guestbook”, “UID”: “36956a55-f48b-4ce4-83b7-da7cf11cdc32”, “kind”: “webapp.zander.com/v1, Kind=Guestbook”, “resource”: {“group”:“webapp.zander.com”,“version”:“v1”,“resource”:“guestbooks”}}
2021-03-11T09:50:29.208Z INFO guestbook-resource validate create {“name”: “guestbook-sample”}
至此,完成了webhook的调用
可以添加业务细节代码啦