nginx ingress最后的倔强: admission webhook

最新推荐文章于 2025-03-12 18:08:34 发布

qingwave

最新推荐文章于 2025-03-12 18:08:34 发布

阅读量8k

点赞数 2

分类专栏：云原生文章标签： kubernetes nginx ingress

本文链接：https://blog.csdn.net/u012986012/article/details/105453459

版权

云原生专栏收录该内容

42 篇文章

订阅专栏

背景

k8s中大多使用nginx-ingress-controller来实现ingress, 但是脆弱的nginx-controller通过ingress解析出nginx配置, 对于某些annotation会reload nignx配置失败, 然后controller就卡死了, 不断重启, 除非删除对应的ingress.

问题复现

创建有问题的ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Host $targethost;
      proxy_buffering     off;
      proxy_pass          http://$targetbackend;
      proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
      proxy_redirect      off;
      proxy_set_header    X-SSL-Client-Verify $ssl_client_verify;
      proxy_set_header    X-SSL-Client-DN $ssl_client_s_dn;
      proxy_set_header    X-Real-IP       $remote_addr;
      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  creationTimestamp: "2020-03-23T04:57:22Z"
  generation: 1
  name: example-ingress
  namespace: kube-system
  resourceVersion: "57681168"
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/ingresses/example-ingress
  uid: c7f66385-6cc2-11ea-b6a8-246e96d4b538
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-svc
          servicePort: 8008
        path: /
  tls:
  - hosts:
    - example.com
    secretName: example-tls
status:
  loadBalancer: {}

查看nginx-ingress-controller状态全部为CrashLoopBackOff

# kubectl get po -n kube-system -owide |grep ingress
nginx-ingress-controller-ftfbg                        1/2     CrashLoopBackOff   6          8m27s
nginx-ingress-controller-hp4pf                        1/2     CrashLoopBackOff   11         24m  
nginx-ingress-controller-qlb4l                        1/2     CrashLoopBackOff   11         24m

查看nginx-ingress-controller日志, 显示reload失败"proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822

-------------------------------------------------------------------------------
W0403 10:26:14.716246       1 queue.go:130] requeuing kube-system/nginx-ingress-controller-4txfk, err 
-------------------------------------------------------------------------------
Error: exit status 1
2020/04/03 10:26:14 [notice] 137#137: ModSecurity-nginx v1.0.0
2020/04/03 10:26:14 [warn] 137#137: duplicate value "error" in /tmp/nginx-cfg911768424:815
nginx: [warn] duplicate value "error" in /tmp/nginx-cfg911768424:815
2020/04/03 10:26:14 [warn] 137#137: duplicate value "timeout" in /tmp/nginx-cfg911768424:815
nginx: [warn] duplicate value "timeout" in /tmp/nginx-cfg911768424:815
2020/04/03 10:26:14 [emerg] 137#137: "proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822
nginx: [emerg] "proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822
nginx: configuration file /tmp/nginx-cfg911768424 test failed

-------------------------------------------------------------------------------
W0403 10:26:16.998897       1 nginx_status.go:207] unexpected error obtaining nginx status info: unexpected error scraping nginx status page: unexpected error scraping nginx : Get http://0.0.0.0:18080/nginx_status: dial tcp 0.0.0.0:18080: connect: connection refused
I0403 10:26:17.526801       1 main.go:167] Received SIGTERM, shutting down
I0403 10:26:17.526827       1 nginx.go:364] Shutting down controller queues
I0403 10:26:17.526845       1 status.go:200] updating status of Ingress rules (remove)
I0403 10:26:17.537511       1 status.go:219] removing address from ingress status ([])
I0403 10:26:17.537593       1 nginx.go:372] Stopping NGINX process
2020/04/03 10:26:17 [notice] 141#141: signal process started
I0403 10:26:20.547669       1 nginx.go:385] NGINX process has stopped
I0403 10:26:20.547692       1 main.go:175] Handled quit, awaiting Pod deletion
I0403 10:26:30.547824       1 main.go:178] Exiting with 0

解决方案

创建一个有问题的ingress, 会影响所有新创建的ingress规则, 又一个集群级别的Bug诞生了.那么有没有办法, 提前检验ingress配置, 有问题就不去reload. 那验证步骤肯定要在请求到达nginx-controller之前来做, 是不是想到了k8s-admission-webhook, 可以在apiserver持久化对象前拦截请求, 去实现自定义的验证规则. 好在新版本的nginx-ingress-controller(v0.25.0+)已经实现了相关的功能, 只需开启对应配置就行.

ApiServer配置

Apiserver开启webhook相关配置, 必须包含MutatingAdmissionWebhook与ValidatingAdmissionWebhook

--admission-control=MutatingAdmissionWebhook,ValidatingAdmissionWebhook

创建webhook相关配置

启用ValidatingAdmissionWebhook必须使用https, 需要配置对应证书

手动生成:

openssl req -x509 -newkey rsa:2048 -keyout certificate.pem -out key.pem -days 365 -nodes -subj "/CN=ingress-validation-webhook.ingress-nginx.svc"

CertificateSigningRequest
通过k8s CertificateSigningRequest来创建(controller-manager需要开启--cluster-signing-cert-file与--cluster-signing-key-file)
可通过如下脚本创建, namespace与service替换成自己的

SERVICE_NAME=ingress-nginx
NAMESPACE=ingress-nginx

TEMP_DIRECTORY=$(mktemp -d)
echo "creating certs in directory ${TEMP_DIRECTORY}"

cat <<EOF >> ${TEMP_DIRECTORY}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${SERVICE_NAME}
DNS.2 = ${SERVICE_NAME}.${NAMESPACE}
DNS.3 = ${SERVICE_NAME}.${NAMESPACE}.svc
EOF

openssl genrsa -out ${TEMP_DIRECTORY}/server-key.pem 2048
openssl req -new -key ${TEMP_DIRECTORY}/server-key.pem \
    -subj "/CN=${SERVICE_NAME}.${NAMESPACE}.svc" \
    -out ${TEMP_DIRECTORY}/server.csr \
    -config ${TEMP_DIRECTORY}/csr.conf

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${SERVICE_NAME}.${NAMESPACE}.svc
spec:
  request: $(cat ${TEMP_DIRECTORY}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

kubectl certificate approve ${SERVICE_NAME}.${NAMESPACE}.svc

for x in $(seq 10); do
    SERVER_CERT=$(kubectl get csr ${SERVICE_NAME}.${NAMESPACE}.svc -o jsonpath='{.status.certificate}')
    if [[ ${SERVER_CERT} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${SERVER_CERT} == '' ]]; then
    echo "ERROR: After approving csr ${SERVICE_NAME}.${NAMESPACE}.svc, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi
echo ${SERVER_CERT} | openssl base64 -d -A -out ${TEMP_DIRECTORY}/server-cert.pem

kubectl create secret generic ingress-nginx.svc \
    --from-file=key.pem=${TEMP_DIRECTORY}/server-key.pem \
    --from-file=cert.pem=${TEMP_DIRECTORY}/server-cert.pem \
    -n ${NAMESPACE}

配置ingress controller

ingress controller需要启用如下参数, 挂载需要的tls证书

flag	description	example usage
`--validating-webhook`	admission webhook的地址	`:8080`
`--validating-webhook-certificate`	webhook证书	`/usr/local/certificates/validating-webhook.pem`
`--validating-webhook-key`	webhook私钥	`/usr/local/certificates/validating-webhook-key.pem`

验证

更新后, 创建有问题的ingress则会拦截, 符合预期

# kubectl apply -f ing.yaml
Error from server: error when creating "ing.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: 
-------------------------------------------------------------------------------
Error: exit status 1
2020/04/02 10:26:04 [emerg] 331#331: directive "proxy_pass" is not terminated by ";" in /tmp/nginx-cfg461116913:2165
nginx: [emerg] directive "proxy_pass" is not terminated by ";" in /tmp/nginx-cfg461116913:2165
nginx: configuration file /tmp/nginx-cfg461116913 test failed

引用

https://kubernetes.io/zh/docs/reference/access-authn-authz/extensible-admission-controllers/
https://kubernetes.github.io/ingress-nginx/deploy/validating-webhook/
https://qingwave.github.io/ingress-nginx-controller-admission-webhook/

更多文章见 Qinng’s Blog