Kubernetes第十曲 调度约束

一、k8s的工作流程图

Kubernetes通过watch的机制进行每个组件的协作，每个组件之间的设计实现了解耦。

流程概述：

控制器放在etcd和scheduler中间

1.创建pod时，会把命令中的pod的属性信息写入到etcd中，记录pod元信息

2.etcd会把信息记录完成的结果告诉api

3.api会把创建pod的告知给scheduler，要求调度分配

4.scheduler会给后端打分，将优先级高的node与pod绑定，bind pod

5.这时会把信息转告给api，node节点IP也知道了，api会把信息写入到etcd，node的地址，pod的节点信息

6.etcd告诉api自己已记录

7.api去调用kubelet，分配dockerIP地址，

8.开始docker run，镜像

9.完成后会反馈给kubelet

10.kubelet会把状态属性告诉api

11.api又将收到的信息写入到etcd中，pod的状态信息

12.etcd反馈已写入

13.再去找kubelet

此时再用kubectl get pods 去找etcd

二、调度方式

nodeName：用于将Pod调度到指定的Node名称上（跳过调度器直接分配）

nodeSelector：用于将Pod调度到匹配Label的Node上

（一）示例1：nodeName

1、编写yaml文件，其中指定nodename

apiVersion: v1  
kind: Pod  
metadata:
  name: pod-example  
  labels:
    app: nginx  
spec:
  nodeName: 192.168.221.90
  containers:
  - name: nginx  
    image: nginx:1.15

2、启动

[root@localhost demo]# kubectl create -f pod5.yaml 
pod/pod-example created
[root@localhost demo]# kubectl get pods
pod-example                       1/1     Running   0          86s
#查看网络
[root@localhost demo]# kubectl get pods -o wide
pod-example                       1/1     Running   0          2m9s   172.17.47.7   192.168.221.90   <none>

3、查看描述信息中的详细事件，没有经过调度器

#查看详细事件（发现未经过调度器）
[root@localhost demo]# kubectl describe pod pod-example

Events:
  Type    Reason   Age   From                      Message
  ----    ------   ----  ----                      -------
  Normal  Pulled   97s   kubelet, 192.168.221.90  Container image "nginx:1.15" already present on machine
  Normal  Created  97s   kubelet, 192.168.19  Created container
  Normal  Started  97s   kubelet, 192.168.195.150  Started container

#清空所有pod
[root@localhost demo]# kubectl delete -f .
[root@localhost demo]# kubectl get pods
No resources found.

（二）示例2：nodeSelector（标签需要经过调度器）

1、使用label标签给节点打标签

#获取标签帮助
[root@localhost demo]# kubectl label --help
Usage:
  kubectl label [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N
[--resource-version=version] [options]
#需要获取node上的NAME名称
[root@localhost demo]# kubectl get node
NAME              STATUS   ROLES    AGE   VERSION
192.168.221.90   Ready    <none>   14d   v1.12.3
192.168.221.100   Ready    <none>   14d   v1.12.3
#给对应的node设置标签分别为kgc=a和kgc=b
[root@localhost demo]# kubectl label nodes 192.168.221.90 kgc=a
node/192.168.221.90 labeled
[root@localhost demo]# kubectl label nodes 192.168.221.100 kgc=b
node/192.168.221.100 labeled

2、查看标签

[root@localhost demo]# kubectl get nodes --show-labels
NAME              STATUS   ROLES    AGE   VERSION   LABELS
192.168.221.90   Ready    <none>   14d   v1.12.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kgc=a,kubernetes.io/hostname=192.168.195.150
192.168.221.100   Ready    <none>   14d   v1.12.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kgc=b,kubernetes.io/hostname=192.168.195.151

3、创建资源

[root@localhost demo]# vim pod5.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-example
  labels:
    app: nginx
spec:
  nodeSelector: 
    kgc: b
  containers:
  - name: nginx
    image: nginx:1.15

[root@localhost demo]# kubectl apply -f pod5.yaml 
pod/pod-example created

4、查看详细事件（通过事件可以观察经过调度器分配）

[root@localhost demo]# kubectl describe pod pod-example
Events:
  Type    Reason     Age   From                      Message
  ----    ------     ----  ----                      -------
  Normal  Scheduled  67s   default-scheduler         Successfully assigned default/pod-example to 192.168.195.151
  Normal  Pulling    66s   kubelet, 192.168.221.90  pulling image "nginx:1.15"
  Normal  Pulled     24s   kubelet, 192.168.221.90  Successfully pulled image "nginx:1.15"
  Normal  Created    24s   kubelet, 192.168.221.90  Created container
  Normal  Started    24s   kubelet, 192.168.221.90  Started container

三、亲和性

3.1、节点亲和性和Pod亲和性

①、节点亲和性

pod.spec.nodeAffinity
  preferredDuringSchedulingIgnoredDuringExecution：软策略
  requiredDuringSchedulingIgnoredDuringExecution：硬策略

②、Pod亲和性

pod.spec.affinity.podAffinity/podAntiAffinity
  preferredDuringSchedulingIgnoredDuringExecution：软策略
  requiredDuringSchedulingIgnoredDuringExecution：硬策略

3.2、键值运算关系

键值	释义
In	label的值在某个列表中
NotIn	label的值不在某个列表中
Gt	label的值大于某个值
Lt	label的值小于某个值
Exists	某个label存在
DoesNotExist	某个label不存在

3.3、示例

#查看节点的标签
[root@master demo]# kubectl  get nodes --show-labels 
NAME     STATUS   ROLES    AGE     VERSION   LABELS
master   Ready    master   6d22h   v1.15.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master,kubernetes.io/os=linux,node-role.kubernetes.io/master=
node01   Ready    <none>   6d22h   v1.15.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux
node02   Ready    <none>   6d22h   v1.15.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node02,kubernetes.io/os=linux

3.3.1、硬策略

[root@master demo]# vim pod1.yaml
apiVersion: v1
kind: Pod
metadata:
  name: affinity
  labels:
    app: node-affinity-pod
spec:
  containers:
  - name: with-node-affinity
    image: soscscs/myapp:v1
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          #指定node的标签
          - key: kubernetes.io/hostname
          #设置Pod安装到kubernetes.io/hostname 的标签值不在valus列表中的node上
            operator: NotIn
            values:
            - node02

[root@master demo]# kubectl  apply  -f pod1.yaml 
pod/affinity created
[root@master demo]# kubectl get pods -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
affinity   1/1     Running   0          9s    10.244.1.94   node01   <none>           <none>

[root@master demo]# vim pod1.yaml
apiVersion: v1
kind: Pod
metadata:
  name: affinity
  labels:
    app: node-affinity-pod
spec:
  containers:
  - name: with-node-affinity
    image: soscscs/myapp:v1
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
         #设置Pod安装到kubernetes.io/hostname的标签值在values列表中的node上
            operator: In
            values:
            #当前调度node没有node03
            - node03

#硬策略不满足条件，Pod 状态一直会处于 Pending 状态。
kubectl delete pod --all && kubectl apply -f pod1.yaml && kubectl get pods -o wide

3.3.2、软策略

[root@master demo]# vim pod2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: affinity
  labels:
    app: node-affinity-pod
spec:
  containers:
  - name: with-node-affinity
    image: soscscs/myapp:v1
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
       #如果有多个软策略选项的话，权重越大，优先级越高
      - weight: 1
        preference:
          matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - node03

[root@master demo]# kubectl  get pods -o wide

[root@master demo]# vim pod2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: affinity
  labels:
    app: node-affinity-pod
spec:
  containers:
  - name: with-node-affinity
    image: soscscs/myapp:v1
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - node02

[root@master demo]# kubectl delete pod --all && kubectl apply -f pod2.yaml && kubectl get pods -o wide

3.3.3、软策略和硬策略一起使用

如果把硬策略和软策略合在一起使用，则要先满足硬策略之后才会满足软策略

apiVersion: v1
kind: Pod
metadata:
  name: affinity
  labels:
    app: node-affinity-pod
spec:
  containers:
  - name: with-node-affinity
    image: soscscs/myapp:v1
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:   #先满足硬策略，排除有kubernetes.io/hostname=node02标签的节点
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: NotIn
            values:
            - node02
      preferredDuringSchedulingIgnoredDuringExecution:  #再满足软策略，优先选择有kgc=a标签的节点
      - weight: 1
        preference:
          matchExpressions:
          - key: apps
            operator: In
            values:
            - a

3.4、亲和性和反亲和

调度策略	匹配标签	操作符	拓扑域支持	调度目标
nodeAffinity	主机	In, NotIn, Exists,DoesNotExist, Gt, Lt	否	指定主机
podAffinit	Pod	In, NotIn, Exists,DoesNotExist	是	Pod与指定Pod同一拓扑域
podAntiAffinity	Pod	In, NotIn, Exists,DoesNotExist	是	Pod与指定Pod不在同一拓扑域

[root@master demo]# vim pod3.yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp01
  labels:
    app: myapp01
spec:
  containers:
  - name: with-node-affinity
    image: soscscs/myapp:v1


[root@master demo]# kubectl  apply  -f pod3.yaml 
pod/myapp01 created

[root@master demo]# kubectl  get pods --show-labels 
NAME       READY   STATUS    RESTARTS   AGE    LABELSd
myapp01    1/1     Running   0          22s    app=myapp01

[root@master demo]# kubectl  get nodes  --show-labels

3.4.1、pod 亲和性

[root@master demo]# vim pod4.yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp02
  labels:
    app: myapp02
spec:
  containers:
  - name: myapp02
    image: soscscs/myapp:v1
  affinity:
    #pod亲和策略
    podAffinity:
      #硬策略
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - myapp01
         #拓扑域
        topologyKey: kubernetes.io/hostname
当存在pod 拥有键app，值为myapp01时,(暂时称这个pod为X），其所在节点的 kubernetes.io/hostname 值，
就成为了新pod调度的依据。（kubernetes.io/hostname 是由字段topologyKey 定义的 ）
新pod调度的节点，其kubernetes.io/hostname值必须和 pod X 所在节点的kubernetes.io/hostname值一致

仅当节点和至少一个已运行且有键为“app”且值为“myapp01”的标签 的 Pod 处于同一拓扑域时，才可以将该 Pod 调度到节点上。 （更确切的说，如果节点 N 具有带有键 kubernetes.io/hostname 和某个值 V 的标签，则 Pod 有资格在节点 N 上运行， 以便集群中至少有一个节点具有键 kubernetes.io/hostname 和值为 V 的节点正在运行具有键“app”和值 “myapp01”的标签的 pod。）

topologyKey 是节点标签的键。如果两个节点使用此键标记并且具有相同的标签值，则调度器会将这两个节点视为处于同一拓扑域中。 调度器试图在每个拓扑域中放置数量均衡的 Pod。

如果 kubernetes.io/hostname 对应的值不一样就是不同的拓扑域。比如 Pod1 在 kubernetes.io/hostname=node01 的 Node 上，Pod2 在 kubernetes.io/hostname=node02 的 Node 上，Pod3 在 kubernetes.io/hostname=node01 的 Node 上，则 Pod2 和 Pod1、Pod3 不在同一个拓扑域，而Pod1 和 Pod3在同一个拓扑域。

[root@master demo]# kubectl  apply  -f pod4.yaml 

[root@master demo]# kubectl get  pods --show-labels -o  wide
NAME       READY   STATUS    RESTARTS   AGE    IP             NODE     NOMINATED NODE   READINESS GATES   LABELS
affinity   1/1     Running   0          122m   10.244.1.100   node01   <none>           <none>            app=node-affinity-pod
myapp01    1/1     Running   0          114m   10.244.2.74    node02   <none>           <none>            app=myapp01
myapp02    1/1     Running   0          46m    10.244.2.75    node02   <none>           <none>            app=myapp02

3.4.2、pod 反亲和性调度

[root@master demo]# vim pod5.yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp03
  labels:
    app: myapp03
spec:
  containers:
  - name: myapp03
    image: soscscs/myapp:v1
  affinity:
    #Pod反亲和
    podAntiAffinity:
      #软策略
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - myapp01
          topologyKey: kubernetes.io/hostname

如果节点处于 Pod 所在的同一拓扑域且具有键“app”和值“myapp01”的标签， 则该 pod 不应将其调度到该节点上。 （如果 topologyKey 为 kubernetes.io/hostname，则意味着当节点和具有键 “app”和值“myapp01”的 Pod 处于相同的区域，Pod 不能被调度到该节点上。）

即：当某个pod拥有键 app，值为myapp01 的标签时（称此pod为X），就不将新pod 调度到和pod X处于 同一个拓扑域的节点（即拥有相同的 topologyKey: kubernetes.io/hostname值 ）

[root@master demo]# kubectl  apply  -f pod5.yaml 
pod/myapp03 created
[root@master demo]# kubectl  get pods --show-labels -o wide
NAME       READY   STATUS    RESTARTS   AGE    IP             NODE     NOMINATED NODE   READINESS GATES   LABELS
affinity   1/1     Running   0          136m   10.244.1.100   node01   <none>           <none>            app=node-affinity-pod
myapp01    1/1     Running   0          128m   10.244.2.74    node02   <none>           <none>            app=myapp01
myapp02    1/1     Running   0          60m    10.244.2.75    node02   <none>           <none>            app=myapp02
myapp03    1/1     Running   0          20s    10.244.1.101   node01   <none>           <none>            app=myapp03

node01 节点的topologyKey值为 kubernetes.io/hostname=node01

node02 节点的 topologyKey值为 kubernetes.io/hostname=node02

因此，node01 和node02是两个拓扑域

而拥有标签 app=myapp01 ，的pod myapp01在node02 节点上。，所以，所有topologyKey值为 kubernetes.io/hostname=node02的节点，都不会被该新pod 选择。

因此，该pod被调度到了 node01上