Kubernetes第十曲 调度约束
一、k8s的工作流程图
Kubernetes通过watch的机制进行每个组件的协作,每个组件之间的设计实现了解耦。
流程概述:
控制器放在etcd和scheduler中间
1.创建pod时,会把命令中的pod的属性信息写入到etcd中,记录pod元信息
2.etcd会把信息记录完成的结果告诉api
3.api会把创建pod的告知给scheduler,要求调度分配
4.scheduler会给后端打分,将优先级高的node与pod绑定,bind pod
5.这时会把信息转告给api,node节点IP也知道了,api会把信息写入到etcd,node的地址,pod的节点信息
6.etcd告诉api自己已记录
7.api去调用kubelet,分配dockerIP地址,
8.开始docker run,镜像
9.完成后会反馈给kubelet
10.kubelet会把状态属性告诉api
11.api又将收到的信息写入到etcd中,pod的状态信息
12.etcd反馈已写入
13.再去找kubelet
此时再用kubectl get pods 去找etcd
二、调度方式
nodeName:用于将Pod调度到指定的Node名称上(跳过调度器直接分配)
nodeSelector:用于将Pod调度到匹配Label的Node上
(一)示例1:nodeName
1、编写yaml文件,其中指定nodename
apiVersion: v1
kind: Pod
metadata:
name: pod-example
labels:
app: nginx
spec:
nodeName: 192.168.221.90
containers:
- name: nginx
image: nginx:1.15
2、启动
[root@localhost demo]# kubectl create -f pod5.yaml
pod/pod-example created
[root@localhost demo]# kubectl get pods
pod-example 1/1 Running 0 86s
#查看网络
[root@localhost demo]# kubectl get pods -o wide
pod-example 1/1 Running 0 2m9s 172.17.47.7 192.168.221.90 <none>
3、查看描述信息中的详细事件,没有经过调度器
#查看详细事件(发现未经过调度器)
[root@localhost demo]# kubectl describe pod pod-example
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 97s kubelet, 192.168.221.90 Container image "nginx:1.15" already present on machine
Normal Created 97s kubelet, 192.168.19 Created container
Normal Started 97s kubelet, 192.168.195.150 Started container
#清空所有pod
[root@localhost demo]# kubectl delete -f .
[root@localhost demo]# kubectl get pods
No resources found.
(二)示例2:nodeSelector(标签需要经过调度器)
1、使用label标签给节点打标签
#获取标签帮助
[root@localhost demo]# kubectl label --help
Usage:
kubectl label [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N
[--resource-version=version] [options]
#需要获取node上的NAME名称
[root@localhost demo]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.221.90 Ready <none> 14d v1.12.3
192.168.221.100 Ready <none> 14d v1.12.3
#给对应的node设置标签分别为kgc=a和kgc=b
[root@localhost demo]# kubectl label nodes 192.168.221.90 kgc=a
node/192.168.221.90 labeled
[root@localhost demo]# kubectl label nodes 192.168.221.100 kgc=b
node/192.168.221.100 labeled
2、查看标签
[root@localhost demo]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
192.168.221.90 Ready <none> 14d v1.12.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kgc=a,kubernetes.io/hostname=192.168.195.150
192.168.221.100 Ready <none> 14d v1.12.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kgc=b,kubernetes.io/hostname=192.168.195.151
3、创建资源
[root@localhost demo]# vim pod5.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-example
labels:
app: nginx
spec:
nodeSelector:
kgc: b
containers:
- name: nginx
image: nginx:1.15
[root@localhost demo]# kubectl apply -f pod5.yaml
pod/pod-example created
4、查看详细事件(通过事件可以观察经过调度器分配)
[root@localhost demo]# kubectl describe pod pod-example
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 67s default-scheduler Successfully assigned default/pod-example to 192.168.195.151
Normal Pulling 66s kubelet, 192.168.221.90 pulling image "nginx:1.15"
Normal Pulled 24s kubelet, 192.168.221.90 Successfully pulled image "nginx:1.15"
Normal Created 24s kubelet, 192.168.221.90 Created container
Normal Started 24s kubelet, 192.168.221.90 Started container
三、亲和性
3.1、节点亲和性和Pod亲和性
①、节点亲和性
pod.spec.nodeAffinity
preferredDuringSchedulingIgnoredDuringExecution:软策略
requiredDuringSchedulingIgnoredDuringExecution:硬策略
②、Pod亲和性
pod.spec.affinity.podAffinity/podAntiAffinity
preferredDuringSchedulingIgnoredDuringExecution:软策略
requiredDuringSchedulingIgnoredDuringExecution:硬策略
3.2、键值运算关系
键值 | 释义 |
---|---|
In | label的值在某个列表中 |
NotIn | label的值不在某个列表中 |
Gt | label的值大于某个值 |
Lt | label的值小于某个值 |
Exists | 某个label存在 |
DoesNotExist | 某个label不存在 |
3.3、示例
#查看节点的标签
[root@master demo]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master Ready master 6d22h v1.15.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master,kubernetes.io/os=linux,node-role.kubernetes.io/master=
node01 Ready <none> 6d22h v1.15.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node01,kubernetes.io/os=linux
node02 Ready <none> 6d22h v1.15.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node02,kubernetes.io/os=linux
3.3.1、硬策略
[root@master demo]# vim pod1.yaml
apiVersion: v1
kind: Pod
metadata:
name: affinity
labels:
app: node-affinity-pod
spec:
containers:
- name: with-node-affinity
image: soscscs/myapp:v1
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
#指定node的标签
- key: kubernetes.io/hostname
#设置Pod安装到kubernetes.io/hostname 的标签值不在valus列表中的node上
operator: NotIn
values:
- node02
[root@master demo]# kubectl apply -f pod1.yaml
pod/affinity created
[root@master demo]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
affinity 1/1 Running 0 9s 10.244.1.94 node01 <none> <none>
[root@master demo]# vim pod1.yaml
apiVersion: v1
kind: Pod
metadata:
name: affinity
labels:
app: node-affinity-pod
spec:
containers:
- name: with-node-affinity
image: soscscs/myapp:v1
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
#设置Pod安装到kubernetes.io/hostname的标签值在values列表中的node上
operator: In
values:
#当前调度node没有node03
- node03
#硬策略不满足条件,Pod 状态一直会处于 Pending 状态。
kubectl delete pod --all && kubectl apply -f pod1.yaml && kubectl get pods -o wide
3.3.2、软策略
[root@master demo]# vim pod2.yaml
apiVersion: v1
kind: Pod
metadata:
name: affinity
labels:
app: node-affinity-pod
spec:
containers:
- name: with-node-affinity
image: soscscs/myapp:v1
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
#如果有多个软策略选项的话,权重越大,优先级越高
- weight: 1
preference:
matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node03
[root@master demo]# kubectl get pods -o wide
[root@master demo]# vim pod2.yaml
apiVersion: v1
kind: Pod
metadata:
name: affinity
labels:
app: node-affinity-pod
spec:
containers:
- name: with-node-affinity
image: soscscs/myapp:v1
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node02
[root@master demo]# kubectl delete pod --all && kubectl apply -f pod2.yaml && kubectl get pods -o wide
3.3.3、软策略和硬策略一起使用
如果把硬策略和软策略合在一起使用,则要先满足硬策略之后才会满足软策略
apiVersion: v1
kind: Pod
metadata:
name: affinity
labels:
app: node-affinity-pod
spec:
containers:
- name: with-node-affinity
image: soscscs/myapp:v1
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution: #先满足硬策略,排除有kubernetes.io/hostname=node02标签的节点
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- node02
preferredDuringSchedulingIgnoredDuringExecution: #再满足软策略,优先选择有kgc=a标签的节点
- weight: 1
preference:
matchExpressions:
- key: apps
operator: In
values:
- a
3.4、亲和性和反亲和
调度策略 | 匹配标签 | 操作符 | 拓扑域支持 | 调度目标 |
---|---|---|---|---|
nodeAffinity | 主机 | In, NotIn, Exists,DoesNotExist, Gt, Lt | 否 | 指定主机 |
podAffinit | Pod | In, NotIn, Exists,DoesNotExist | 是 | Pod与指定Pod同一拓扑域 |
podAntiAffinity | Pod | In, NotIn, Exists,DoesNotExist | 是 | Pod与指定Pod不在同一拓扑域 |
[root@master demo]# vim pod3.yaml
apiVersion: v1
kind: Pod
metadata:
name: myapp01
labels:
app: myapp01
spec:
containers:
- name: with-node-affinity
image: soscscs/myapp:v1
[root@master demo]# kubectl apply -f pod3.yaml
pod/myapp01 created
[root@master demo]# kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELSd
myapp01 1/1 Running 0 22s app=myapp01
[root@master demo]# kubectl get nodes --show-labels
3.4.1、pod 亲和性
[root@master demo]# vim pod4.yaml
apiVersion: v1
kind: Pod
metadata:
name: myapp02
labels:
app: myapp02
spec:
containers:
- name: myapp02
image: soscscs/myapp:v1
affinity:
#pod亲和策略
podAffinity:
#硬策略
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- myapp01
#拓扑域
topologyKey: kubernetes.io/hostname
当存在pod 拥有键app,值为myapp01时,(暂时称这个pod为X),其所在节点的 kubernetes.io/hostname 值,
就成为了新pod调度的依据。(kubernetes.io/hostname 是由字段topologyKey 定义的 )
新pod调度的节点,其kubernetes.io/hostname值必须和 pod X 所在节点的kubernetes.io/hostname值一致
仅当节点和至少一个已运行且有键为“app”且值为“myapp01”的标签 的 Pod 处于同一拓扑域时,才可以将该 Pod 调度到节点上。 (更确切的说,如果节点 N 具有带有键 kubernetes.io/hostname 和某个值 V 的标签,则 Pod 有资格在节点 N 上运行, 以便集群中至少有一个节点具有键 kubernetes.io/hostname 和值为 V 的节点正在运行具有键“app”和值 “myapp01”的标签的 pod。)
topologyKey 是节点标签的键。如果两个节点使用此键标记并且具有相同的标签值,则调度器会将这两个节点视为处于同一拓扑域中。 调度器试图在每个拓扑域中放置数量均衡的 Pod。
如果 kubernetes.io/hostname 对应的值不一样就是不同的拓扑域。比如 Pod1 在 kubernetes.io/hostname=node01 的 Node 上,Pod2 在 kubernetes.io/hostname=node02 的 Node 上,Pod3 在 kubernetes.io/hostname=node01 的 Node 上,则 Pod2 和 Pod1、Pod3 不在同一个拓扑域,而Pod1 和 Pod3在同一个拓扑域。
[root@master demo]# kubectl apply -f pod4.yaml
[root@master demo]# kubectl get pods --show-labels -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
affinity 1/1 Running 0 122m 10.244.1.100 node01 <none> <none> app=node-affinity-pod
myapp01 1/1 Running 0 114m 10.244.2.74 node02 <none> <none> app=myapp01
myapp02 1/1 Running 0 46m 10.244.2.75 node02 <none> <none> app=myapp02
3.4.2、pod 反亲和性调度
[root@master demo]# vim pod5.yaml
apiVersion: v1
kind: Pod
metadata:
name: myapp03
labels:
app: myapp03
spec:
containers:
- name: myapp03
image: soscscs/myapp:v1
affinity:
#Pod反亲和
podAntiAffinity:
#软策略
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- myapp01
topologyKey: kubernetes.io/hostname
如果节点处于 Pod 所在的同一拓扑域且具有键“app”和值“myapp01”的标签, 则该 pod 不应将其调度到该节点上。 (如果 topologyKey 为 kubernetes.io/hostname,则意味着当节点和具有键 “app”和值“myapp01”的 Pod 处于相同的区域,Pod 不能被调度到该节点上。)
即:当某个pod拥有键 app,值为myapp01 的标签时(称此pod为X),就不将新pod 调度到和pod X处于 同一个拓扑域的节点(即拥有相同的 topologyKey: kubernetes.io/hostname值 )
[root@master demo]# kubectl apply -f pod5.yaml
pod/myapp03 created
[root@master demo]# kubectl get pods --show-labels -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
affinity 1/1 Running 0 136m 10.244.1.100 node01 <none> <none> app=node-affinity-pod
myapp01 1/1 Running 0 128m 10.244.2.74 node02 <none> <none> app=myapp01
myapp02 1/1 Running 0 60m 10.244.2.75 node02 <none> <none> app=myapp02
myapp03 1/1 Running 0 20s 10.244.1.101 node01 <none> <none> app=myapp03
node01 节点的topologyKey值为 kubernetes.io/hostname=node01
node02 节点的 topologyKey值为 kubernetes.io/hostname=node02
因此,node01 和node02是两个拓扑域
而拥有标签 app=myapp01 ,的pod myapp01在node02 节点上。,所以,所有topologyKey值为 kubernetes.io/hostname=node02的节点,都不会被该新pod 选择。
因此,该pod被调度到了 node01上