networkPlugin cni failed to teardown pod nginx network: invalid version \“\“: the version is empty“

现象

某些nginx的pod一直在Terminating状态 执行kubectl describe po <pod名称> -n <命名空间名称>例如kubectl describe po nginx-deployment-demo-675745c9cd-wtxbj -n demo发现报错

Events:
  Type     Reason         Age                  From     Message
  ----     ------         ----                 ----     -------
  Warning  FailedKillPod  15h (x442 over 16h)  kubelet  error killing pod: failed to "KillPodSandbox" for "8e085d2c-4608-4d74-95b6-de44befff0fe" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"nginx-deployment-demo-675745c9cd-wtxbj_demo\" network: invalid version \"\": the version is empty"

分析

报错信息很明确说明了是网络插件版本号的问题, 我用的是flannel插件, 修改configmap中的cni-conf.json
在cbr0 这一行上面新增一行"cniVersion": "0.3.1", 版本号根据自己实际情况修改,我是参考 https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm 无包管理器的情况这一栏安装的CNI 插件, "cniVersion" 有 "0.1.0", "0.2.0", "0.3.0", "0.3.1"这四个版本,详情请查看 https://github.com/containernetworking/cni/blob/v0.6.0-rc1/SPEC.md

修改过程如下:

• 执行kubectl get cm -A 查看configmap列表找到flannel插件对应的configmap,

kubectl get cm -A    
NAMESPACE         NAME                                 DATA   AGE
default           kube-root-ca.crt                     1      4d17h
kube-node-lease   kube-root-ca.crt                     1      4d17h
kube-public       cluster-info                         1      4d17h
kube-public       kube-root-ca.crt                     1      4d17h
kube-system       coredns                              1      4d17h
kube-system       extension-apiserver-authentication   6      4d17h
kube-system       kube-flannel-cfg                     2      4d16h
kube-system       kube-proxy                           2      4d17h
kube-system       kube-root-ca.crt                     1      4d17h
kube-system       kubeadm-config                       1      4d17h
kube-system       kubelet-config-1.22                  1      4d17h

• 执行kubectl edit cm <configmap名称> -n <命名空间名称> 修改configmap
  例如 : kubectl edit cm kube-flannel-cfg -n kube-system
  修改flannel插件对应的configmap下面的cni-conf.json并在cbr0 这一行上面新增一行"cniVersion": "0.3.1",
  修改/etc/cni/net.d 下面的配置文件例如 /etc/cni/net.d/10-flannel.conflist, 新增一行"cniVersion": "0.3.1", 如下

{
  "cniVersion": "0.3.1",
  "name": "cbr0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

然后在各节点执行systemctl restart kubelet.service 重启服务生效

参考

https://www.cnblogs.com/winstom/p/11836844.html
https://github.com/aws/amazon-vpc-cni-k8s/issues/1412
https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm
https://github.com/containernetworking/cni/blob/v0.6.0-rc1/SPEC.md