kubernetes webhook 本地调试避坑指南

本文着重介绍本人在测试webhook过程中遇到的问题，及最终解决方案。
我在linux机器上测试，安装kubebuilder,kustomize等问题不再赘述。
一，安装项目
1.首先在gopath下新建目录，并进入目录

mkdir mywebhook
cd mywebhook/

2.初始化项目

[root@host mywebhook]# kubebuilder init --domain zander.com

3，生成kind,crd

kubebuilder create api --group webapp --version v1 --kind Guestbook

4.安装crd

make install

5 生成cr

kubectl apply -f config/samples/

6.生成webhook

kubebuilder create webhook --group webapp --version v1 --kind Guestbook --programmatic-validation --defaulting

一切到这里都还非常顺利，下面将进入一步一个坑的阶段

二，启动服务
1.证书问题

make run

报错，找不到k8s_webhook-server/serving-certs/tls.crt

解决办法

cp /etc/kubernetes/pki/apiserver.crt /tmp/k8s_webhook-server/serving-certs/tls.crt
cp /etc/kubernetes/pki/apiserver.key /tmp/k8s_webhook-server/serving-certs/tls.key

如果本地没有这个证书，也可以使用工具生成

然后make run可以正常启动了

2.webhook掉不到
这是由于apiserver根本就不知道有webhook
解决办法，修改config/default/kustomization.yaml文件
将webhook相关配置打开，如下

# Adds namespace to all resources.
namespace: mywebhook-system

# Value of this field is prepended to the
# names of all resources, e.g. a deployment named
# "wordpress" becomes "alices-wordpress".
# Note that it should also match with the prefix (text before '-') of the namespace
# field above.
namePrefix: mywebhook-

# Labels to add to all resources and selectors.
#commonLabels:
#  someName: someValue

bases:
- ../crd
- ../rbac
- ../manager
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in 
# crd/kustomization.yaml
- ../webhook
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
- ../certmanager
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'. 
#- ../prometheus

patchesStrategicMerge:
  # Protect the /metrics endpoint by putting it behind auth.
  # If you want your controller-manager to expose the /metrics
  # endpoint w/o any authn/z, please comment the following line.
- manager_auth_proxy_patch.yaml

# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in 
# crd/kustomization.yaml
- manager_webhook_patch.yaml

# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
# 'CERTMANAGER' needs to be enabled to use ca injection
- webhookcainjection_patch.yaml

# the following config is for teaching kustomize how to do var substitution
vars:
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
  objref:
    kind: Certificate
    group: cert-manager.io
    version: v1alpha2
    name: serving-cert # this name should match the one in certificate.yaml
  fieldref:
    fieldpath: metadata.namespace
- name: CERTIFICATE_NAME
  objref:
    kind: Certificate
    group: cert-manager.io
    version: v1alpha2
    name: serving-cert # this name should match the one in certificate.yaml
- name: SERVICE_NAMESPACE # namespace of the service
  objref:
    kind: Service
    version: v1
    name: webhook-service
  fieldref:
    fieldpath: metadata.namespace
- name: SERVICE_NAME
  objref:
    kind: Service
    version: v1
    name: webhook-service

3.生成controller镜像

make docker-build

报错go: github.com/go-logr/logr@v0.1.0: Get https://proxy.golang.org/github.com/go-logr/logr/@v/v0.1.0.mod: dial tcp 216.58.200.49:443: connect: connection refused

解决办法，修改镜像
将Dockerfile修改如下：

FROM registry.xxx/baseimages/centos:centos7.5.1804
WORKDIR /
COPY ./bin/manager .
ENTRYPOINT ["/manager"]

4.部署controller等

make deploy

查看容器状态

kubectl get pod -A

发现容器挂掉，镜像拉取失败

mywebhook-controller-manager-67f555b9bf-qf455 0/2
ErrImagePull

查看细节

kubectl logs -f mywebhook-controller-manager-67f555b9bf-qf455 -n mywebhook-system manager

发现报错

Error from server (BadRequest): container “manager” in pod “mywebhook-controller-manager-67f555b9bf-qf455” is waiting to start: trying and failing to pull image

说明是镜像拉取失败，由于我使用本地镜像，并没有docker-push到远程仓库，所以拉取失败
解决办法
将docker 镜像版本号修改，不使用latest ，以避免从远程拉取
修改makefile

IMG ?= guestbook-controller:1.0.0
manager: generate fmt vet manifests
	go build -o bin/manager main.go
deploy: manager
	cd config/manager && kustomize edit set image controller=${IMG}
	kustomize build config/default | kubectl apply -f -

修改config/manager/manager.yaml

image: guestbook-controller:1.0.0

删除原先部署的内容，再执行make docker-build,make deploy
发现容器起来了

mywebhook-system mywebhook-controller-manager-698cc75d79-9t2zh 1/1 Running 0 21s

查看下日志

[root@host mywebhook]# kubectl logs -f mywebhook-controller-manager-698cc75d79-9t2zh -n mywebhook-system manager
2021-03-11T09:47:06.630Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-03-11T09:47:06.646Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "webapp.zander.com/v1, Kind=Guestbook", "path": "/mutate-webapp-zander-com-v1-guestbook"}
2021-03-11T09:47:06.646Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/mutate-webapp-zander-com-v1-guestbook"}
2021-03-11T09:47:06.646Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "webapp.zander.com/v1, Kind=Guestbook", "path": "/validate-webapp-zander-com-v1-guestbook"}
2021-03-11T09:47:06.646Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/validate-webapp-zander-com-v1-guestbook"}
2021-03-11T09:47:06.646Z	INFO	setup	starting manager
2021-03-11T09:47:06.733Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
I0311 09:47:06.762562       1 leaderelection.go:242] attempting to acquire leader lease  mywebhook-system/7117d994.zander.com...
2021-03-11T09:47:06.794Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-03-11T09:47:08.287Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2021-03-11T09:47:08.288Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 9443}
2021-03-11T09:47:08.307Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
I0311 09:47:12.954688       1 leaderelection.go:252] successfully acquired lease mywebhook-system/7117d994.zander.com
2021-03-11T09:47:13.371Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "guestbook", "source": "kind source: /, Kind="}
2021-03-11T09:47:13.273Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"mywebhook-system","name":"7117d994.zander.com","uid":"e25380ed-3bb6-422f-9231-b2182538e84e","apiVersion":"v1","resourceVersion":"18235607"}, "reason": "LeaderElection", "message": "mywebhook-controller-manager-698cc75d79-9t2zh_a5599b20-7d36-4adf-8f4b-21124531e831 became leader"}
2021-03-11T09:47:18.740Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "guestbook"}
2021-03-11T09:47:18.813Z	INFO	controller-runtime.controller	Starting workers	{"controller": "guestbook", "worker count": 1}

可以看到webhook服务已经启动了

修改webapp_v1_guestbook.yaml
中参数，重新应用

 kubectl apply -f config/samples/webapp_v1_guestbook.yaml 

再看controller日志，发现webhook的日志打印出来了！

NFO guestbook-resource default {“name”: “guestbook-sample”}
2021-03-11T09:50:28.617Z DEBUG controller-runtime.webhook.webhooks wrote response {“webhook”: “/mutate-webapp-zander-com-v1-guestbook”, “UID”: “24beb268-c42c-42c9-989c-1901882b7b6f”, “allowed”: true, “result”: {}, “resultError”: “got runtime.Object without object metadata: &Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:200,}”}
2021-03-11T09:50:29.045Z DEBUG controller-runtime.webhook.webhooks received request {“webhook”: “/validate-webapp-zander-com-v1-guestbook”, “UID”: “36956a55-f48b-4ce4-83b7-da7cf11cdc32”, “kind”: “webapp.zander.com/v1, Kind=Guestbook”, “resource”: {“group”:“webapp.zander.com”,“version”:“v1”,“resource”:“guestbooks”}}
2021-03-11T09:50:29.208Z INFO guestbook-resource validate create {“name”: “guestbook-sample”}

至此，完成了webhook的调用
可以添加业务细节代码啦