现象
某些nginx的pod一直在Terminating
状态 执行kubectl describe po <pod名称> -n <命名空间名称>
例如kubectl describe po nginx-deployment-demo-675745c9cd-wtxbj -n demo
发现报错
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedKillPod 15h (x442 over 16h) kubelet error killing pod: failed to "KillPodSandbox" for "8e085d2c-4608-4d74-95b6-de44befff0fe" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"nginx-deployment-demo-675745c9cd-wtxbj_demo\" network: invalid version \"\": the version is empty"
分析
报错信息很明确说明了是网络插件版本号的问题, 我用的是flannel
插件, 修改configmap
中的cni-conf.json
在cbr0
这一行上面新增一行"cniVersion": "0.3.1",
版本号根据自己实际情况修改,我是参考 https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm 无包管理器的情况
这一栏安装的CNI 插件, "cniVersion"
有 "0.1.0", "0.2.0", "0.3.0", "0.3.1"
这四个版本,详情请查看 https://github.com/containernetworking/cni/blob/v0.6.0-rc1/SPEC.md
修改过程如下:
- 执行
kubectl get cm -A
查看configmap
列表找到flannel
插件对应的configmap
,
kubectl get cm -A
NAMESPACE NAME DATA AGE
default kube-root-ca.crt 1 4d17h
kube-node-lease kube-root-ca.crt 1 4d17h
kube-public cluster-info 1 4d17h
kube-public kube-root-ca.crt 1 4d17h
kube-system coredns 1 4d17h
kube-system extension-apiserver-authentication 6 4d17h
kube-system kube-flannel-cfg 2 4d16h
kube-system kube-proxy 2 4d17h
kube-system kube-root-ca.crt 1 4d17h
kube-system kubeadm-config 1 4d17h
kube-system kubelet-config-1.22 1 4d17h
- 执行
kubectl edit cm <configmap名称> -n <命名空间名称>
修改configmap
例如 :kubectl edit cm kube-flannel-cfg -n kube-system
修改flannel
插件对应的configmap
下面的cni-conf.json
并在cbr0
这一行上面新增一行"cniVersion": "0.3.1",
修改/etc/cni/net.d
下面的配置文件例如/etc/cni/net.d/10-flannel.conflist
, 新增一行"cniVersion": "0.3.1",
如下
{
"cniVersion": "0.3.1",
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
然后在各节点执行systemctl restart kubelet.service
重启服务生效
参考
https://www.cnblogs.com/winstom/p/11836844.html
https://github.com/aws/amazon-vpc-cni-k8s/issues/1412
https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm
https://github.com/containernetworking/cni/blob/v0.6.0-rc1/SPEC.md