k8s容器限速

背景

因业务需求，需要对 K8S 集群中 Pod 的出入带宽进行限制。
原理就是在pod启动的时候，在对应的虚拟网络设备加上相应的tc规则，通过tc实现的流量限制。

方案实现

编辑配置文件/etc/cni/net.d/10-calico.conflist，修改配置参数，启用bandwidth插件（默认是开启状态）

sudo vim /etc/cni/net.d/10-calico.conflist

{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "log_file_path": "/var/log/calico/cni/cni.log",
      "datastore_type": "kubernetes",
      "nodename": "test-cq-worker-1",
      "mtu": 1440,
      "ipam": {
          "type": "calico-ipam"
      },
      "policy": {
          "type": "k8s"
      },
      "kubernetes": {
          "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      }
    },
    {
      "type": "portmap",
      "snat": true,
      "capabilities": {"portMappings": true}
    },
    {
      "type": "bandwidth",         #####添加带宽限速参数
      "capabilities": {"bandwidth": true}
    }
  ]
  

重启一下 kubelet

systemctl restart kubelet 

实验一：测试在没有流量限制的情况下，两个pod之间的传输速率。

通过下面的yaml启动两个pod：

---
apiVersion: v1
kind: Pod
metadata:
  name: perf1
  labels:
    app: perf1
#   annotations:
#     kubernetes.io/ingress-bandwidth: 10M       ###限制进带宽为10M
#     kubernetes.io/egress-bandwidth: 10M        ###限制出带宽为10M
spec:
  containers:
  - name: perf-server
    image: registry.sail-auk.io:8082/elynn/pperf:latest       ###含有iperf3命令的镜像即可
    imagePullPolicy: IfNotPresent      ###镜像的拉取策略
    command:
    - "/opt/runserver.sh"
    ports:
    - containerPort: 5201
    - containerPort: 5203

---
apiVersion: v1
kind: Pod
metadata:
  name: perf2
  labels:
    app: perf2
#  annotations:
#    kubernetes.io/ingress-bandwidth: 1M
#    kubernetes.io/egress-bandwidth: 1M
spec:
  containers:
  - name: perf-server
    image: registry.sail-auk.io:8082/elynn/pperf:latest
    imagePullPolicy: IfNotPresent
    command:
    - "/opt/runserver.sh"
    ports:
    - containerPort: 5201
    - containerPort: 5203

检查到容器已running，进入运行的容器perf1

执行iperf3 -c 10.0.222.20 -t 10
测试单机转发流量，可以看到在不限流的情况下，流量可以达到万兆

实验二：测试有流量限制的情况下，两个pod之间的传输速率。

删除上面的两个测试容器，并修改启动pod的配置参数，打开下列三行参数：

  annotations:
    kubernetes.io/ingress-bandwidth: 10M
    kubernetes.io/egress-bandwidth: 10M

完整yaml文件如下：

---
apiVersion: v1
kind: Pod
metadata:
  name: perf1
  labels:
    app: perf1
  annotations:
    kubernetes.io/ingress-bandwidth: 10M
    kubernetes.io/egress-bandwidth: 10M
spec:
  containers:
  - name: perf-server
    image: registry.sail-auk.io:8082/elynn/pperf:latest
    imagePullPolicy: IfNotPresent
    command:
    - "/opt/runserver.sh"
    ports:
    - containerPort: 5201
    - containerPort: 5203

---
apiVersion: v1
kind: Pod
metadata:
  name: perf2
  labels:
    app: perf2
#  annotations:
#    kubernetes.io/ingress-bandwidth: 1M
#    kubernetes.io/egress-bandwidth: 1M
spec:
  containers:
  - name: perf-server
    image: registry.sail-auk.io:8082/elynn/pperf:latest
    imagePullPolicy: IfNotPresent
    command:
    - "/opt/runserver.sh"
    ports:
    - containerPort: 5201
    - containerPort: 5203

启动pod，创建成功之后进入运行的容器perf1

执行iperf3 -c 10.0.222.59 -t 10
测试限速后的单机转发流量，可以看到流量被限制到了10Mbits/s。