【k8s admission学习】yaml 部署注意的坑

总览

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admission-registry-sa

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: admission-registry-clusterrole
rules:
- verbs: ["*"]
  resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]
  apiGroups: ["admissionregistration.k8s.io"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admission-registry-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admission-registry-clusterrole
subjects:
  - kind: ServiceAccount
    name: admission-registry-sa
    namespace: default

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-registry
  labels:
    app: admission-registry
spec:
  selector:
    matchLabels:
      app: admission-registry
  template:
    metadata:
      labels:
        app: admission-registry
    spec:
      serviceAccountName: admission-registry-sa
      initContainers:
      - name: webhook-init
        image: dfy007/admission-registry-tls:v1.8
        imagePullPolicy: IfNotPresent
        env:
        - name: WEBHOOK_NAMESPACE
          value: default
        - name: WEBHOOK_SERVICE
          value: admission-registry
        - name: VALIDATE_CONFIG
          value: admission-registry
        - name: VALIDATE_PATH
          value: /validate
        - name: MUTATE_CONFIG
          value: admission-registry-mutate
        - name: MUTATE_PATH
          value: /mutate
        volumeMounts:
        - name: webhook-certs
          mountPath: /etc/webhook/certs
      containers:
      - name: webhook
        image: dfy007/admission-webhook-example:v1.8
        imagePullPolicy: IfNotPresent
        env:
        - name: WHITELIST_REGISTRIES
          value: "docker.io,gcr.io"
        ports:
        - containerPort: 443
        volumeMounts:
        - name: webhook-certs
          mountPath: /etc/webhook/certs
          readOnly: true
      volumes:
      - name: webhook-certs
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: admission-registry
  labels:
    app: admission-registry
spec:
  ports:
    - port: 443
      targetPort: 443
  selector:
    app: admission-registry

1. 赋予权限 —— RBAC

• 通过 initContainer 我们将自建的 ca 证书赋予到 validatingwebhookconfigurations和mutatingwebhookconfigurations的caBundle中，用于实现对于 webhook 提供的 server 的证书的验证
  • 因此需要具有创建 validatingwebhookconfigurations和mutatingwebhookconfigurations 资源的权限
  • 赋予权限通过 RBAC 来实现

坑

1. ServiceAccount 名称要保持一致
   • 之前手敲，打错了，结果排错时，找了好久
2. validatingwebhookconfigurations和mutatingwebhookconfigurations 要注意后面的s
   • 同样之前，缺少了s，导致mutatingwebhookconfigurations无法创建，排错了好久

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admission-registry-sa

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: admission-registry-clusterrole
rules:
- verbs: ["*"]
  # 注意资源后面的 小 s
  resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]
  apiGroups: ["admissionregistration.k8s.io"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admission-registry-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admission-registry-clusterrole
subjects:
  - kind: ServiceAccount
    name: admission-registry-sa  # 与上面 ServiceAccount 名称保持一致
    namespace: default

2. 注意 ServiceAccount 的挂载

前面我们只创建了 ServiceAccount，并赋予了权限，但是我们创建 Deployment 时，若没有携带这个 ServiceAccount，就会采用默认的 default ServiceAccount，导致没有权限创建（同时花费时间排错）

apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-registry
  labels:
    app: admission-registry
spec:
  selector:
    matchLabels:
      app: admission-registry
  template:
    metadata:
      labels:
        app: admission-registry   # label  要与下面创建的 Service 对应
    spec:
      # 注意此处 ServiceAccount 的挂载，与上面保持一致
      # 若没有挂载 就会采用默认的 default ServiceAccount 导致权限不足
      serviceAccountName: admission-registry-sa
      initContainers:
      - name: webhook-init
        image: dfy007/admission-registry-tls:v1.8
        imagePullPolicy: IfNotPresent
        env:
        - name: WEBHOOK_NAMESPACE
          value: default
        - name: WEBHOOK_SERVICE
          value: admission-registry
        - name: VALIDATE_CONFIG
          value: admission-registry
        - name: VALIDATE_PATH
          value: /validate
        - name: MUTATE_CONFIG
          value: admission-registry-mutate
        - name: MUTATE_PATH
          value: /mutate
        volumeMounts:
        - name: webhook-certs
          mountPath: /etc/webhook/certs
      containers:
      - name: webhook
        image: dfy007/admission-webhook-example:v1.8
        imagePullPolicy: IfNotPresent
        env:
        - name: WHITELIST_REGISTRIES
          value: "docker.io,gcr.io"
        ports:
        - containerPort: 443  # 暴露的端口  要与下面创建的 Service 对应
        volumeMounts:
        - name: webhook-certs
          mountPath: /etc/webhook/certs
          readOnly: true
      volumes:
      - name: webhook-certs
        emptyDir: {}

3. Webhook Service 的创建

1. 前面我们运行的 Deployment 和 Pod 就是 证书创建 和 webhook 后端
2. 那么 Apiserver 如何能访问到呢，这就需要了 k8s Service（此处也是之前创建 validatingwebhookconfigurations和mutatingwebhookconfigurations 注入进去的参数）
3. 所以要记得创建 Service 关联对应的 webhook 后端 Pod

apiVersion: v1
kind: Service
metadata:
  name: admission-registry
  labels:
    app: admission-registry  
spec:
  ports:
    - port: 443
      targetPort: 443        # # 要与上面 webhook pod 的 port 对应
  selector:
    app: admission-registry  # 要与上面 webhook pod 的 label 对应

4. admissionregistration 示例

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  creationTimestamp: "2022-10-07T01:57:10Z"
  generation: 1
  name: admission-registry
  resourceVersion: "32755"
  uid: b6058e85-b9d3-4f18-9ae5-a9770c384079
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
  # 防止 ca 证书用于验证server证书
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZnekNDQTJ1Z0F3SUJBZ0lDQitVd0RRWUpLb1pJaHZjTkFRRUxCUUF3VXpFTE1Ba0dBMVVFQmhNQ1EwNHgKRURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RHpBTkJnTlZCQW9UQm1SbQplUzVwYnpFUE1BMEdBMVVFQ3hNR1pHWjVMbWx2TUI0WERUSXlNVEF3TnpBeE5UY3dPRm9YRFRNeU1UQXdOekF4Ck5UY3dPRm93VXpFTE1Ba0dBMVVFQmhNQ1EwNHhFREFPQmdOVkJBZ1RCMEpsYVdwcGJtY3hFREFPQmdOVkJBY1QKQjBKbGFXcHBibWN4RHpBTkJnTlZCQW9UQm1SbWVTNXBiekVQTUEwR0ExVUVDeE1HWkdaNUxtbHZNSUlDSWpBTgpCZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUF1eGpDMElLSDNVdklvV1pBZTROY2pOSlphd1BoCm5Pc0Q5Y09xQzV1V0tnQlBpamdxWXB2bUJTamcrWVh0MXRDMk00WEI1aDlQdVJaV01KdW9ZcTlhVnZpTC9pM2YKa3Myck1leStKMkNMY1ZvcmdlNC82SHVXNy9VSUtMTlhrbkNLK0FZUis5dDVkTVFHZHR6YXpsNlBXS0IvUlpvSgpyNk0yODdHQU1vNW1zcnZ0WEN3WDhKOTlRKzVWd3lGb20veFZmR0dtMHU0dXBRQkl4VzVUM2JuVE5CSlc2eVZhCkt4VmRwMFVmb0hTczRNem1QVHJlbm5MQ2dzZmpsbkhUa1YxNGo0d3hyZXVVQkhrZmZHcjI1V0NDTlN0eGRSazMKaXRiZ2FtSFppa3U4bTdGTlg4SlJ3Qml0VDZ2cHg4aXFaSURybEFucW5vbUxVZWFRdW9jUTZhRWtWS3A0UDFIOApQSlg5bnlXRlRpREt3Q3ZSVVJXRTVKZm1wcmJPRElnakdnczQ4V2NnMTFXWDBLOW1pN05YQ3dKQm1NTmFvU09pCjZIVlRqaDMwV2Q1TCtPM0p4T3hjUHc0T2hPSWowL3FmR0VkelpyZXFnSTBVUm9STW8zY2JZNFZiSGpNM1VDNWoKOEdETVo0VWVZNnMrWnVWSTlKclJDSE5zVkdZNVdlcWV0SXBwWXhpcGxoQ1RmbTFFVUllYUZ3YUVLSjBGVjZhaQp1QzIzTWZEYnRWZng0UWxCRkZ0R252bjBoTUNVWDFiV2ZHZDNxMXF0UEZ1VE9yMnRoUno5Wk5nQmRtUHdhT3ZjCnFLdGdmMGRhRm5vVkM2dUMxc0tOL2JnNDZ3Y1BBcm1hY1Q4dXR1c1Z4a3dOOW1jUkFwQVpnQUVFRUMzaDJoVGkKNXl0cjRBVUVVWDIyZGo4Q0F3RUFBYU5oTUY4d0RnWURWUjBQQVFIL0JBUURBZ0tFTUIwR0ExVWRKUVFXTUJRRwpDQ3NHQVFVRkJ3TUNCZ2dyQmdFRkJRY0RBVEFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlRWCk1yOVJJWXJYeHRvakZjVkFVbTJMbzNjQWtUQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FnRUFUdFo2UHhYYVI2Z28Kc2lIYk5tdmszNTg0WlEzT0ZURnJoYzJ2ak9ONnN1TElnSWd5M3RFQ3NjRks5VzNRRFN5RWVNMWdCNFpRQ2pldAo4N0ZETVhZblF1djFzdmpkNlk1ajF3VUJPZnErZjI0clFLaEg4UzdyS0F5SjZNdkgzU3gybUtJaWh4eFpFZHR4CnBaS3AwV3Z4Wkgxb1hNL2hhL2RQQWx6UmdjbG5HcmQ5dGgyYm9vNUhKSXJvZy9uUG5PSDE2SzlhdHdSYVgySXgKbldFQTdwWk01dU9HYzJMVWlHdXJ5Zm15VEhuMGZDS3I1ckJJRkwyVWU2eGlGRHlKR1Y4ZXdEazhSWmdQcWZMSgpZellVYkFvdTFXVWRRU2hWL2FUWElDdHo4SHc4blFHTTdtT1JNRkphOGQ5dE42RHBBcDZPOVk3WHkyUkhuK1lwClR2SlpYUEdpQUhnOFhlOFNHcmpUeHZCeit3ZlBQcjVOK1V6cWx4S3c1OU9HUUtGSHUwcjdsTUcxQXIrK09tVVEKd3lDYnphdVNsUU5vN0ZoUG54TG5NUzlWbTNpazJ5SmNuUC9rMXF1Y2d2L055cUdEcHJPZUsxRXlWMWRiVUpiZApDRWtGenoycHdWbjM5aStuOTBVa0NsLzRsWEtPckYxZ3locGFhL2s0VHc1UHlWQ2o2OWtIZ3JhYjRzVjVST2x0CnYxcE5EcTl6bWd0TVUyOTk0T3o5YUtBanlsOUphWC9EbGVlSHhNeEw1clFuMzhTaks1TUplOXcwb0FjVHptT2sKUW5tVURlemRTaGlSOTRtbk9ycW42MHRHbCt1ZHNBdExuM2srOGtUTVBEM1o1a3BPUWNERTQ4WUxJdzlEQXhlQwpUbHZld1g3UTJQZFRVanQ5MjR1cytUZkFVcXY0TDFnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    service:
      name: admission-registry
      namespace: default
      path: /validate
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: io.dfy.admission-registry
  namespaceSelector: {}
  objectSelector: {}
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10