pod拓扑传播限制和svc拓扑感知

pod拓扑传播限制

k8在1.19版本之前，需要配置apiserver、scheduler启用该功能。配置feature gate开启EvenPodsSpread

• maxSkew

描述Pod可能不均匀分布的程度。它是给定拓扑类型的任何两个拓扑域中的匹配Pod数量之间的最大允许差异。它必须大于零。其语义根据以下值的不同而有所不同whenUnsatisfiable：

当whenUnsatisfiable等于“ DoNotSchedule”时，maxSkew是目标拓扑中匹配的容器数与全局最小值之间的最大允许差值。
当whenUnsatisfiable等于“ ScheduleAnyway”时，调度程序会为拓扑提供更高的优先级，这将有助于减少时滞。

• topologyKey

是节点标签的键。如果两个节点都用此密钥标记并且具有相同的值，则调度程序会将两个节点都视为处于同一拓扑中。调度程序尝试将均衡数量的Pod放入每个拓扑域中。

• whenUnsatisfiable

表示如果Pod不满足传播约束，则如何处理：

DoNotSchedule （默认）告诉调度程序不要调度它。
ScheduleAnyway 告诉调度程序在对节点进行优先级排序以最大程度地减少偏斜时仍要调度它。

• labelSelector

用于查找匹配的Pod。计算与该标签选择器匹配的Pod，以确定其相应拓扑域中的Pod数。有关更多详细信息，请参见标签选择器。

示例

• 部署文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    server: nginx
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      server: nginx
      app: web
  template:
    metadata:
      name: nginx
      labels:
        server: nginx
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              server: nginx
      containers:
      - name: nginx
        image: nginx:1.14
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    server: nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP

部署结果

[root@master demo]# kubectl get pods -owide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
nginx-668d778d75-d9zj8   0/1     Pending   0          10s   <none>        <none>   <none>           <none>
nginx-668d778d75-wlnrh   1/1     Running   0          10s   10.244.1.33   slave1   <none>           <none>
[root@master demo]# kubectl describe pods nginx-668d778d75-d9zj8 | grep -A 5 Event
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  0/2 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/2 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) had taints that the pod didn't tolerate.

svc拓扑感知

svc的拓扑感知使用EndpointSlice进行控制，在开启EndpointSlice后，将会替换endpoint。此特性为1.16版本的新增功能，在1.17版本，此特性在进入beta，但是仅默认开启了API，未启用EndpointSlice controller 和kube-proxy对EndpointSlices的使用。目前有效的拓扑键仅限于：

kubernetes.io/hostname、topology.kubernetes.io/zone、topology.kubernetes.io/region

也可自定义标签，不过自定义标签不会自动添加至endpointslices，需要手动进行添加。

感知流程（C–>S）

1. 获取C端所在的hostname、zone、region
2. 访问、解析svc，路由至endpointlices
3. 按照svc的感知顺序在endpointlices中进行匹配
4. 找到匹配域后，流量将会被路由至该匹配域的pod
5. 找不到匹配域后，流量将会被丢弃

示例

[root@master demo]# kubectl get endpointslices.discovery.k8s.io nginx-dpnmv -oyaml
addressType: IPv4
apiVersion: discovery.k8s.io/v1beta1
endpoints:
- addresses:
  - 10.244.0.17    #可路由pod地址
  conditions:      #pod状态
    ready: true    
  targetRef:       #绑定的pod信息
    kind: Pod
    name: nginx-668d778d75-nvscq
    namespace: default
    resourceVersion: "38950"
    uid: 08d86026-c149-4a53-b54f-f212eec2218f
  topology:        #感知拓扑域，svc的感知拓扑域需与这里进行对应。比如这里拓扑域仅包含hostname，则svc添加zone拓扑域将会无效。
    kubernetes.io/hostname: master
- addresses:
  - 10.244.1.35
  conditions:
    ready: false
  targetRef:
    kind: Pod
    name: nginx-5cb8d5d999-lkxms
    namespace: default
    resourceVersion: "42226"
    uid: 090b8eb4-aaef-48fb-bc11-3ce878434578
  topology:
    kubernetes.io/hostname: slave1
kind: EndpointSlice
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2021-01-13T06:47:27Z"
  creationTimestamp: "2021-01-13T06:18:26Z"
  generateName: nginx-
  generation: 19
  labels:
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
    kubernetes.io/service-name: nginx
  name: nginx-dpnmv
  namespace: default
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Service
    name: nginx
    uid: e3a1fa54-3b1f-4a0d-bec3-cf53b1a89ec4
  resourceVersion: "42227"
  selfLink: /apis/discovery.k8s.io/v1beta1/namespaces/default/endpointslices/nginx-dpnmv
  uid: 06c5b3b2-8865-4aab-ab7e-407941d0fbe6
ports:
- name: http
  port: 80
  protocol: TCP

前提条件

• kubernetes 1.17或者更高
• kube-proxy使用iptables或IPVS
• 启用EndpointSlice

启用EndpointSlice

• apiserver

    - --feature-gates=EvenPodsSpread=true,ServiceTopology=true,EndpointSlice=true

• controller-manager

    - --feature-gates=ServiceTopology=true,EndpointSlice=true

• scheduler

    - --feature-gates=EvenPodsSpread=true,ServiceTopology=true,EndpointSlice=true

• kube-proxy

修改configmap

    featureGates:
      EndpointSlice: true
      ServiceTopology: true

示例

• 部署client、nginx

client、nginx均为2副本，集群的2个节点各自调度1个副本，分别使用client的2个副本调用nginx服务

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    server: nginx
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      server: nginx
      app: web
  template:
    metadata:
      name: nginx
      labels:
        server: nginx
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              server: nginx
      containers:
      - name: nginx
        image: nginx:1.14
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  topologyKeys: ["kubernetes.io/hostname", "topology.kubernetes.io/zone", "topology.kubernetes.io/region", '*']
  selector:
    server: nginx
  ports: 
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP

[root@master demo]# kubectl get nodes --show-labels 
NAME     STATUS   ROLES    AGE     VERSION   LABELS
master   Ready    master   19h     v1.17.0   topology.kubernetes.io/zone=beijing,kubernetes.io/hostname=master
slave1   Ready    <none>   4h15m   v1.17.0   topology.kubernetes.io/zone=tianjin,kubernetes.io/hostname=slave1

[root@master demo]# kubectl get pods -owide
NAME                      READY   STATUS    RESTARTS   AGE     IP            NODE     NOMINATED NODE   READINESS GATES
client-85c4d985f6-4mbtv   1/1     Running   0          156m    10.244.1.31   slave1   <none>           <none>
client-85c4d985f6-7gvk7   1/1     Running   0          32m     10.244.0.15   master   <none>           <none>
nginx-668d778d75-fx5kc    1/1     Running   0          5m28s   10.244.1.32   slave1   <none>           <none>
nginx-668d778d75-whrld    1/1     Running   0          162m    10.244.0.14   master   <none>           <none>

• master节点client调用nginx

[root@master demo]# kubectl exec -it client-85c4d985f6-7gvk7 bash
root@client-85c4d985f6-7gvk7:/usr/local/tomcat# for i in {1..5};do curl nginx ; done
beijing
beijing
beijing
beijing
beijing

• slave1节点client调用nginx

[root@master demo]# kubectl exec -it client-85c4d985f6-4mbtv bash
root@client-85c4d985f6-4mbtv:/usr/local/tomcat# for i in {1..5};do curl nginx ; done
tianjin
tianjin
tianjin
tianjin
tianjin