K8s Taint 和 Tolerations

用途

强隔离，独占节点。如按部门或业务划拨一批节点，不让其他用户的Pod调度到这个节点上，只有加上了特定的Toleration才能调度到这个Taint 节点上，其他的统统不行。

为什么不用Label隔离呢？
Label不是强制性的，且即使打了Label，其他用户的Pod还是可以调度上去。除非要求所有其他用户在所有部署中都增加反亲和性来实现，这显然是不合适的。所以最好的方式就是我们自己给自己独占的节点设置好Taint，别人不需要改动什么。

Taint 种类：

1. NoSchedule: 新的Pod 不调度到该Node上，不影响正在运行的Pod；
2. PreferNoSchedule: 新的Pod尽量不调度到该Node上；
3. NoExecute: 新的Pod不调度到该Node上，并且驱逐已经在运行的Pod。Pod可以有一个tolerationSeconds, 定义多久之后驱逐。

实验

准备2个节点的集群，节点名分别为i-cd6fjvxn、i-jgdj1gg0。

Taint Node

kubectl taint nodes i-cd6fjvxn  for-special-user=cadmin:NoSchedule
node/i-cd6fjvxn tainted

kubectl taint nodes i-jgdj1gg0  for-special-user=cadmin:NoSchedule
node/i-jgdj1gg0 tainted

先给两个Node 都加上Taint。

没有 Toleration 的部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      imagePullSecrets: 
      - name: kleven-registry
      containers:
        - name: nginx
          image: kleven-repository:8083/nginx

部署后会发现：当没有设置toleration的时候，无法完成调度，Pod一直处于pending状态。

kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-586dd79555-8nztl   0/1     Pending   0          37s

describe命令查看原因：Pod没有对应的toleration与Node 的 Taint的匹配。

kubectl describe pod nginx-deployment-586dd79555-8nztl

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  67s   default-scheduler  0/2 nodes are available: 2 node(s) had taint {for-special-user: cadmin}, that the pod didn't tolerate.

我们将其中一个节点的Taint删掉

kubectl taint nodes i-jgdj1gg0 for-special-user=cadmin:NoSchedule-
node/i-jgdj1gg0 untainted

这时就会发现Pod已经调度成功

kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-586dd79555-8nztl   1/1     Running   0          4m55s

查看description

Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  2m16s (x4 over 5m40s)  default-scheduler  0/2 nodes are available: 2 node(s) had taint {for-special-user: cadmin}, that the pod didn't tolerate.
  Normal   Scheduled         111s                   default-scheduler  Successfully assigned default/nginx-deployment-586dd79555-8nztl to i-jgdj1gg0
  Normal   Pulling           110s                   kubelet            Pulling image "kleven-repository:8083/nginx"
  Normal   Pulled            110s                   kubelet            Successfully pulled image "kleven-repository:8083/nginx" in 70.61121ms
  Normal   Created           110s                   kubelet            Created container nginx
  Normal   Started           110s                   kubelet            Started container nginx

总结：当一个Node标记有Taint的时候，如果Pod没有与之配置的toleration的话，是不会被调度上去的。

包含 Toleration 的部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: app
                  operator: In
                  values:
                    - nginx
            topologyKey: kubernetes.io/hostname
      imagePullSecrets: 
      - name: kleven-registry
      containers:
        - name: nginx
          image: kleven-repository:8083/nginx
      tolerations:
        - key: "for-special-user"
          operator: "Equal"
          value: "cadmin"
          effect: "NoSchedule"

调度成功，同时这里使用了反亲和性将2个副本调度到了不同的节点上。

kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP                NODE         NOMINATED NODE   READINESS GATES
nginx-deployment-5d46d59f85-krzwz   1/1     Running   0          12s   192.168.235.210   i-cd6fjvxn   <none>           <none>
nginx-deployment-5d46d59f85-z29nw   1/1     Running   0          12s   192.168.2.143     i-jgdj1gg0   <none>           <none>

总结： 带有Toleration的Pod可以调度到与之匹配的Taint节点上，也可以调度到没有Taint的节点上。