k8s 自动清理完成的 job

需求：自动清理完成的 job

[root@iZbp deploy]# kubectl get node -o wide
NAME                       STATUS   ROLES    AGE   VERSION             INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                         KERNEL-VERSION            CONTAINER-RUNTIME
cn-hangzhou.172.24.4.71     Ready    <none>   87d   v1.20.11-aliyun.1   172.24.4.71     <none>        Alibaba Cloud Linux (Aliyun Linux) 2.1903 LTS (Hunting Beagle)   4.19.91-24.1.al7.x86_64   docker://19.3.15
cn-hangzhou.172.24.4.81     Ready    <none>   87d   v1.20.11-aliyun.1   172.24.4.81     <none>        Alibaba Cloud Linux (Aliyun Linux) 2.1903 LTS (Hunting Beagle)   4.19.91-24.1.al7.x86_64   docker://19.3.15
cn-hangzhou.172.24.4.154   Ready    <none>   87d   v1.20.11-aliyun.1   172.24.4.154   <none>        Alibaba Cloud Linux (Aliyun Linux) 2.1903 LTS (Hunting Beagle)   4.19.91-24.1.al7.x86_64   docker://19.3.15
[root@iZbp deploy]#

阿里云的k8s版本信息

[root@iZbp deploy]# kubectl version
Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-aliyun.1", GitCommit:"88d5e8b", GitTreeState:"", BuildDate:"2021-10-25T13:55:31Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-aliyun.1", GitCommit:"d966b365d87fb2783deab0d244047d340f1be146", GitTreeState:"clean", BuildDate:"2022-01-19T01:10:01Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
[root@iZbp10t1lvw1hrjeukfmsyZ deploy]# kubelet --version
Kubernetes v1.20.11-aliyun.1
[root@iZbp deploy]#

注意：1.22.3-aliyun.1 版本的集群测试是支持的 1.20.11-aliyun.1 集群版本不支持 Job 的 .spec.ttlSecondsAfterFinished 字段

所以采用以下的办法实现

代码下载地址

https://github.com/hjacobs/kube-job-cleaner

查看 job 和 cronjob 的版本

[root@iZbp ~]# kubectl api-resources | grep job
cronjobs                          cj               batch/v1beta1                          true         CronJob
jobs                                               batch/v1                               true         Job
[root@iZbp ~]#

rbac.yaml

[root@iZbp deploy]# cat /root/kube-job-cleaner/deploy/rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-job-cleaner
  namespace: kube-job-cleaner

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-job-cleaner-clusterrolebinding
subjects:
- kind: ServiceAccount
  name: kube-job-cleaner
  namespace: kube-job-cleaner
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: ""
[root@iZbp deploy]#

cronjob.yaml

[root@iZbp deploy]# cat /root/kube-job-cleaner/deploy/cronjob.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: kube-job-cleaner
  namespace: kube-job-cleaner
  labels:
    application: kube-job-cleaner
    version: "0.1"
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            application: kube-job-cleaner
            version: "0.1"
        spec:
          serviceAccountName: kube-job-cleaner
          restartPolicy: OnFailure
          containers:
          - name: cleaner
            command: ["/cleaner.py","--seconds","10","--namespace","opt-tool"]
            image: "registry.test.cn/exchange/hjacobs/kube-job-cleaner:0.8-2-g5f9c3ab-dirty"
            resources:
              limits:
                cpu: 200m
                memory: 100Mi
              requests:
                cpu: 50m
                memory: 50Mi
          tolerations:
          - effect: NoSchedule
            key: key
            operator: Equal
            value: xsj

部署

kubectl create ns kube-job-cleaner
kubectl apply -f deploy/
# 如果等不急，可以手动出发cronjob
kubectl create job xyz --from=cronjob/kube-job-cleaner -n opt-tool

查看

[root@iZbp ~]# kubectl get cronjob -n kube-job-cleaner
NAME               SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
kube-job-cleaner   * * * * *   False     1        18s             14h
[root@iZbp10t1lvw1hrjeukfmsyZ ~]# kubectl get job -n kube-job-cleaner
NAME                          COMPLETIONS   DURATION   AGE
kube-job-cleaner-1646361420   1/1           3s         2m16s
kube-job-cleaner-1646361480   1/1           3s         76s
kube-job-cleaner-1646361540   1/1           4s         16s
[root@iZbp ~]#

测试

[root@iZbp ~]# cat test-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: test-job
  namespace: opt-tool
spec:
  ttlSecondsAfterFinished: 0
  manualSelector: true
  completions: 6
  parallelism: 3
  selector:
    matchLabels:
      app: counter-pod
  template:
    metadata:
      labels:
        app: counter-pod
    spec:
      restartPolicy: Never
      containers:
      - name: counter
        image: busybox:1.30
        command: ["bin/sh","-c","for i in 3 2 1; do echo $i;sleep 3;done"]
      tolerations:
      - effect: NoSchedule
        key: key
        operator: Equal
        value: xsj
[root@iZbp ~]#

窗口1：试验结果

[root@iZbp ~]# kubectl apply -f test-job.yaml
job.batch/test-job created
[root@iZbp ~]# kubectl get pod -n opt-tool -w
NAME                                          READY   STATUS    RESTARTS   AGE
airflow-6f76598ddf-r666r                      2/2     Running   0          3d14h
database-mysql-0                              1/1     Running   0          3d15h
test-job-khs8j                                1/1     Running   0          8s
test-job-n9qmc                                1/1     Running   0          8s
test-job-zqm9b                                1/1     Running   0          8s
test-job-zqm9b                                0/1     Completed   0          11s
test-job-2z7d5                                0/1     Pending     0          0s
test-job-2z7d5                                0/1     Pending     0          0s
test-job-khs8j                                0/1     Completed   0          11s
test-job-22xnz                                0/1     Pending     0          0s
test-job-22xnz                                0/1     Pending     0          0s
test-job-n9qmc                                0/1     Completed   0          11s
test-job-2b85t                                0/1     Pending     0          0s
test-job-2z7d5                                0/1     ContainerCreating   0          0s
test-job-2b85t                                0/1     Pending             0          0s
test-job-22xnz                                0/1     ContainerCreating   0          0s
test-job-2b85t                                0/1     ContainerCreating   0          0s
test-job-2z7d5                                1/1     Running             0          1s
test-job-2b85t                                1/1     Running             0          1s
test-job-22xnz                                1/1     Running             0          1s
test-job-22xnz                                0/1     Completed           0          10s
test-job-2z7d5                                0/1     Completed           0          10s
test-job-2b85t                                0/1     Completed           0          10s

窗口2：pod 跑完就自动删除了

[root@iZbp ~]# kubectl get pod -n opt-tool
NAME                                          READY   STATUS    RESTARTS   AGE
airflow-6f76598ddf-r666r                      2/2     Running   0          3d15h
database-mysql-0                              1/1     Running   0          3d15h
[root@iZbp ~]#