pod拓扑传播限制
k8在1.19版本之前,需要配置apiserver、scheduler启用该功能。配置feature gate开启EvenPodsSpread
- maxSkew
描述Pod可能不均匀分布的程度。它是给定拓扑类型的任何两个拓扑域中的匹配Pod数量之间的最大允许差异。它必须大于零。其语义根据以下值的不同而有所不同whenUnsatisfiable:
当whenUnsatisfiable等于“ DoNotSchedule”时,maxSkew是目标拓扑中匹配的容器数与全局最小值之间的最大允许差值。
当whenUnsatisfiable等于“ ScheduleAnyway”时,调度程序会为拓扑提供更高的优先级,这将有助于减少时滞。
- topologyKey
是节点标签的键。如果两个节点都用此密钥标记并且具有相同的值,则调度程序会将两个节点都视为处于同一拓扑中。调度程序尝试将均衡数量的Pod放入每个拓扑域中。
- whenUnsatisfiable
表示如果Pod不满足传播约束,则如何处理:
DoNotSchedule (默认)告诉调度程序不要调度它。
ScheduleAnyway 告诉调度程序在对节点进行优先级排序以最大程度地减少偏斜时仍要调度它。
- labelSelector
用于查找匹配的Pod。计算与该标签选择器匹配的Pod,以确定其相应拓扑域中的Pod数。有关更多详细信息,请参见标签选择器。
示例
- 部署文件
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
server: nginx
app: web
spec:
replicas: 2
selector:
matchLabels:
server: nginx
app: web
template:
metadata:
name: nginx
labels:
server: nginx
app: web
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
server: nginx
containers:
- name: nginx
image: nginx:1.14
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
server: nginx
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
部署结果
[root@master demo]# kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-668d778d75-d9zj8 0/1 Pending 0 10s <none> <none> <none> <none>
nginx-668d778d75-wlnrh 1/1 Running 0 10s 10.244.1.33 slave1 <none> <none>
[root@master demo]# kubectl describe pods nginx-668d778d75-d9zj8 | grep -A 5 Event
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler 0/2 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling <unknown> default-scheduler 0/2 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) had taints that the pod didn't tolerate.
svc拓扑感知
svc的拓扑感知使用EndpointSlice进行控制,在开启EndpointSlice后,将会替换endpoint。此特性为1.16版本的新增功能,在1.17版本,此特性在进入beta,但是仅默认开启了API,未启用EndpointSlice controller 和kube-proxy对EndpointSlices的使用。目前有效的拓扑键仅限于:
kubernetes.io/hostname
、topology.kubernetes.io/zone
、topology.kubernetes.io/region
也可自定义标签,不过自定义标签不会自动添加至endpointslices,需要手动进行添加。
感知流程(C–>S)
- 获取C端所在的hostname、zone、region
- 访问、解析svc,路由至endpointlices
- 按照svc的感知顺序在endpointlices中进行匹配
- 找到匹配域后,流量将会被路由至该匹配域的pod
- 找不到匹配域后,流量将会被丢弃
示例
[root@master demo]# kubectl get endpointslices.discovery.k8s.io nginx-dpnmv -oyaml
addressType: IPv4
apiVersion: discovery.k8s.io/v1beta1
endpoints:
- addresses:
- 10.244.0.17 #可路由pod地址
conditions: #pod状态
ready: true
targetRef: #绑定的pod信息
kind: Pod
name: nginx-668d778d75-nvscq
namespace: default
resourceVersion: "38950"
uid: 08d86026-c149-4a53-b54f-f212eec2218f
topology: #感知拓扑域,svc的感知拓扑域需与这里进行对应。比如这里拓扑域仅包含hostname,则svc添加zone拓扑域将会无效。
kubernetes.io/hostname: master
- addresses:
- 10.244.1.35
conditions:
ready: false
targetRef:
kind: Pod
name: nginx-5cb8d5d999-lkxms
namespace: default
resourceVersion: "42226"
uid: 090b8eb4-aaef-48fb-bc11-3ce878434578
topology:
kubernetes.io/hostname: slave1
kind: EndpointSlice
metadata:
annotations:
endpoints.kubernetes.io/last-change-trigger-time: "2021-01-13T06:47:27Z"
creationTimestamp: "2021-01-13T06:18:26Z"
generateName: nginx-
generation: 19
labels:
endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
kubernetes.io/service-name: nginx
name: nginx-dpnmv
namespace: default
ownerReferences:
- apiVersion: v1
blockOwnerDeletion: true
controller: true
kind: Service
name: nginx
uid: e3a1fa54-3b1f-4a0d-bec3-cf53b1a89ec4
resourceVersion: "42227"
selfLink: /apis/discovery.k8s.io/v1beta1/namespaces/default/endpointslices/nginx-dpnmv
uid: 06c5b3b2-8865-4aab-ab7e-407941d0fbe6
ports:
- name: http
port: 80
protocol: TCP
前提条件
- kubernetes 1.17或者更高
- kube-proxy使用iptables或IPVS
- 启用EndpointSlice
启用EndpointSlice
- apiserver
- --feature-gates=EvenPodsSpread=true,ServiceTopology=true,EndpointSlice=true
- controller-manager
- --feature-gates=ServiceTopology=true,EndpointSlice=true
- scheduler
- --feature-gates=EvenPodsSpread=true,ServiceTopology=true,EndpointSlice=true
- kube-proxy
修改configmap
featureGates:
EndpointSlice: true
ServiceTopology: true
示例
- 部署client、nginx
client、nginx均为2副本,集群的2个节点各自调度1个副本,分别使用client的2个副本调用nginx服务
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
server: nginx
app: web
spec:
replicas: 2
selector:
matchLabels:
server: nginx
app: web
template:
metadata:
name: nginx
labels:
server: nginx
app: web
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
server: nginx
containers:
- name: nginx
image: nginx:1.14
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
topologyKeys: ["kubernetes.io/hostname", "topology.kubernetes.io/zone", "topology.kubernetes.io/region", '*']
selector:
server: nginx
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
[root@master demo]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master Ready master 19h v1.17.0 topology.kubernetes.io/zone=beijing,kubernetes.io/hostname=master
slave1 Ready <none> 4h15m v1.17.0 topology.kubernetes.io/zone=tianjin,kubernetes.io/hostname=slave1
[root@master demo]# kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
client-85c4d985f6-4mbtv 1/1 Running 0 156m 10.244.1.31 slave1 <none> <none>
client-85c4d985f6-7gvk7 1/1 Running 0 32m 10.244.0.15 master <none> <none>
nginx-668d778d75-fx5kc 1/1 Running 0 5m28s 10.244.1.32 slave1 <none> <none>
nginx-668d778d75-whrld 1/1 Running 0 162m 10.244.0.14 master <none> <none>
- master节点client调用nginx
[root@master demo]# kubectl exec -it client-85c4d985f6-7gvk7 bash
root@client-85c4d985f6-7gvk7:/usr/local/tomcat# for i in {1..5};do curl nginx ; done
beijing
beijing
beijing
beijing
beijing
- slave1节点client调用nginx
[root@master demo]# kubectl exec -it client-85c4d985f6-4mbtv bash
root@client-85c4d985f6-4mbtv:/usr/local/tomcat# for i in {1..5};do curl nginx ; done
tianjin
tianjin
tianjin
tianjin
tianjin