nginx ingress最后的倔强: admission webhook

背景

k8s中大多使用nginx-ingress-controller来实现ingress, 但是脆弱的nginx-controller通过ingress解析出nginx配置, 对于某些annotation会reload nignx配置失败, 然后controller就卡死了, 不断重启, 除非删除对应的ingress.

问题复现

创建有问题的ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Host $targethost;
      proxy_buffering     off;
      proxy_pass          http://$targetbackend;
      proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
      proxy_redirect      off;
      proxy_set_header    X-SSL-Client-Verify $ssl_client_verify;
      proxy_set_header    X-SSL-Client-DN $ssl_client_s_dn;
      proxy_set_header    X-Real-IP       $remote_addr;
      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
  creationTimestamp: "2020-03-23T04:57:22Z"
  generation: 1
  name: example-ingress
  namespace: kube-system
  resourceVersion: "57681168"
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/ingresses/example-ingress
  uid: c7f66385-6cc2-11ea-b6a8-246e96d4b538
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-svc
          servicePort: 8008
        path: /
  tls:
  - hosts:
    - example.com
    secretName: example-tls
status:
  loadBalancer: {}

查看nginx-ingress-controller状态全部为CrashLoopBackOff

# kubectl get po -n kube-system -owide |grep ingress
nginx-ingress-controller-ftfbg                        1/2     CrashLoopBackOff   6          8m27s
nginx-ingress-controller-hp4pf                        1/2     CrashLoopBackOff   11         24m  
nginx-ingress-controller-qlb4l                        1/2     CrashLoopBackOff   11         24m   

查看nginx-ingress-controller日志, 显示reload失败"proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822

-------------------------------------------------------------------------------
W0403 10:26:14.716246       1 queue.go:130] requeuing kube-system/nginx-ingress-controller-4txfk, err 
-------------------------------------------------------------------------------
Error: exit status 1
2020/04/03 10:26:14 [notice] 137#137: ModSecurity-nginx v1.0.0
2020/04/03 10:26:14 [warn] 137#137: duplicate value "error" in /tmp/nginx-cfg911768424:815
nginx: [warn] duplicate value "error" in /tmp/nginx-cfg911768424:815
2020/04/03 10:26:14 [warn] 137#137: duplicate value "timeout" in /tmp/nginx-cfg911768424:815
nginx: [warn] duplicate value "timeout" in /tmp/nginx-cfg911768424:815
2020/04/03 10:26:14 [emerg] 137#137: "proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822
nginx: [emerg] "proxy_pass" directive is duplicate in /tmp/nginx-cfg911768424:822
nginx: configuration file /tmp/nginx-cfg911768424 test failed

-------------------------------------------------------------------------------
W0403 10:26:16.998897       1 nginx_status.go:207] unexpected error obtaining nginx status info: unexpected error scraping nginx status page: unexpected error scraping nginx : Get http://0.0.0.0:18080/nginx_status: dial tcp 0.0.0.0:18080: connect: connection refused
I0403 10:26:17.526801       1 main.go:167] Received SIGTERM, shutting down
I0403 10:26:17.526827       1 nginx.go:364] Shutting down controller queues
I0403 10:26:17.526845       1 status.go:200] updating status of Ingress rules (remove)
I0403 10:26:17.537511       1 status.go:219] removing address from ingress status ([])
I0403 10:26:17.537593       1 nginx.go:372] Stopping NGINX process
2020/04/03 10:26:17 [notice] 141#141: signal process started
I0403 10:26:20.547669       1 nginx.go:385] NGINX process has stopped
I0403 10:26:20.547692       1 main.go:175] Handled quit, awaiting Pod deletion
I0403 10:26:30.547824       1 main.go:178] Exiting with 0

解决方案

创建一个有问题的ingress, 会影响所有新创建的ingress规则, 又一个集群级别的Bug诞生了.那么有没有办法, 提前检验ingress配置, 有问题就不去reload. 那验证步骤肯定要在请求到达nginx-controller之前来做, 是不是想到了k8s-admission-webhook, 可以在apiserver持久化对象前拦截请求, 去实现自定义的验证规则. 好在新版本的nginx-ingress-controller(v0.25.0+)已经实现了相关的功能, 只需开启对应配置就行.

ApiServer配置

Apiserver开启webhook相关配置, 必须包含MutatingAdmissionWebhook与ValidatingAdmissionWebhook

--admission-control=MutatingAdmissionWebhook,ValidatingAdmissionWebhook

创建webhook相关配置

启用ValidatingAdmissionWebhook必须使用https, 需要配置对应证书

• 手动生成:

  openssl req -x509 -newkey rsa:2048 -keyout certificate.pem -out key.pem -days 365 -nodes -subj "/CN=ingress-validation-webhook.ingress-nginx.svc"
  

• CertificateSigningRequest
  通过k8s CertificateSigningRequest来创建(controller-manager需要开启--cluster-signing-cert-file与--cluster-signing-key-file)
  可通过如下脚本创建, namespace与service替换成自己的

  SERVICE_NAME=ingress-nginx
  NAMESPACE=ingress-nginx
  
  TEMP_DIRECTORY=$(mktemp -d)
  echo "creating certs in directory ${TEMP_DIRECTORY}"
  
  cat <<EOF >> ${TEMP_DIRECTORY}/csr.conf
  [req]
  req_extensions = v3_req
  distinguished_name = req_distinguished_name
  [req_distinguished_name]
  [ v3_req ]
  basicConstraints = CA:FALSE
  keyUsage = nonRepudiation, digitalSignature, keyEncipherment
  extendedKeyUsage = serverAuth
  subjectAltName = @alt_names
  [alt_names]
  DNS.1 = ${SERVICE_NAME}
  DNS.2 = ${SERVICE_NAME}.${NAMESPACE}
  DNS.3 = ${SERVICE_NAME}.${NAMESPACE}.svc
  EOF
  
  openssl genrsa -out ${TEMP_DIRECTORY}/server-key.pem 2048
  openssl req -new -key ${TEMP_DIRECTORY}/server-key.pem \
      -subj "/CN=${SERVICE_NAME}.${NAMESPACE}.svc" \
      -out ${TEMP_DIRECTORY}/server.csr \
      -config ${TEMP_DIRECTORY}/csr.conf
  
  cat <<EOF | kubectl create -f -
  apiVersion: certificates.k8s.io/v1beta1
  kind: CertificateSigningRequest
  metadata:
    name: ${SERVICE_NAME}.${NAMESPACE}.svc
  spec:
    request: $(cat ${TEMP_DIRECTORY}/server.csr | base64 | tr -d '\n')
    usages:
    - digital signature
    - key encipherment
    - server auth
  EOF
  
  kubectl certificate approve ${SERVICE_NAME}.${NAMESPACE}.svc
  
  for x in $(seq 10); do
      SERVER_CERT=$(kubectl get csr ${SERVICE_NAME}.${NAMESPACE}.svc -o jsonpath='{.status.certificate}')
      if [[ ${SERVER_CERT} != '' ]]; then
          break
      fi
      sleep 1
  done
  if [[ ${SERVER_CERT} == '' ]]; then
      echo "ERROR: After approving csr ${SERVICE_NAME}.${NAMESPACE}.svc, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
      exit 1
  fi
  echo ${SERVER_CERT} | openssl base64 -d -A -out ${TEMP_DIRECTORY}/server-cert.pem
  
  kubectl create secret generic ingress-nginx.svc \
      --from-file=key.pem=${TEMP_DIRECTORY}/server-key.pem \
      --from-file=cert.pem=${TEMP_DIRECTORY}/server-cert.pem \
      -n ${NAMESPACE}
  

配置ingress controller

ingress controller需要启用如下参数, 挂载需要的tls证书

flag	description	example usage
--validating-webhook	admission webhook的地址	:8080
--validating-webhook-certificate	webhook证书	/usr/local/certificates/validating-webhook.pem
--validating-webhook-key	webhook私钥	/usr/local/certificates/validating-webhook-key.pem

验证

更新后, 创建有问题的ingress则会拦截, 符合预期

# kubectl apply -f ing.yaml
Error from server: error when creating "ing.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: 
-------------------------------------------------------------------------------
Error: exit status 1
2020/04/02 10:26:04 [emerg] 331#331: directive "proxy_pass" is not terminated by ";" in /tmp/nginx-cfg461116913:2165
nginx: [emerg] directive "proxy_pass" is not terminated by ";" in /tmp/nginx-cfg461116913:2165
nginx: configuration file /tmp/nginx-cfg461116913 test failed

引用

• https://kubernetes.io/zh/docs/reference/access-authn-authz/extensible-admission-controllers/
• https://kubernetes.github.io/ingress-nginx/deploy/validating-webhook/
• https://qingwave.github.io/ingress-nginx-controller-admission-webhook/

---

更多文章见 Qinng’s Blog