1.问题描述
在k8s master get nodes时,报如下错误
命令:
kubectl get nodes
......
E0408 15:31:50.400954 23612 memcache.go:265] couldn't get current server API group list: Get "https://192.168.10.236:6443/api?timeout=20s": dial tcp 192.168.10.236:6443: connect: connection refused
......
查看kubelet服务状态,发现服务处于loaded状态,输入如下信息:
命令:
systemctl status kubelet
......
...Unit kubelet.service entered failed state.
...kubelet.service failed.
......
查看kubelet日志,有如下报错
命令:
journalctl -xefu kubelet
......
Oct 13 16:00:15 master kubelet[27951]: E0408 16:00:15.786948 27951 run.go:74] "command failed" err="failed to parse kubelet flag: unknown flag: --network-plugin"
......
2.原因分析
根据日志初步分析,是cni网络模块出了问题,kubelet重启后,导致启动网络插件的命令无法执行。
进一步在master和minion节点上查看kubelet的版本,发现了问题
master节点
命令:
rpm -qa |grep kube
[root@centos79-3 kubernetes]# rpm -qa |grep kube
kubectl-1.24.9-0.x86_64
kubernetes-cni-1.24.9-0.x86_64
kubelet-1.24.9-0.x86_64
kubeadm-1.24.9-0.x86_64
[root@centos79-3 kubernetes]#
minion节点
命令:
rpm -qa |grep kube
[root@cent79-2 ~]# rpm -qa | grep kube
kubectl-1.20.4-0.x86_64
kubernetes-cni-1.2.0-0.x86_64
kubelet-1.20.4-0.x86_64
kubeadm-1.20.4-0.x86_64
[root@cent79-2 ~]#
即master节点的kubeadm、kubelet、kubectl、kubernetes-cni版本和minion节点上kubeadm、kubelet、kubectl、kubernetes-cni版本不一致,且master节点使用的版本高于minion节点。需要将master节点的kubeadm、kubelet、kubectl、kubernetes-cni版本降级至与minion节点一致
3.解决方案
-->master节点的kubeadm、kubelet、kubectl、kubernetes-cni版本执行降级
命令:
yum list kubeadm --showduplicates | sort -r
yum downgrade --setopt=obsoletes=0 -y kubectl-1.20.4-0.x86_64 kubernetes-cni-1.2.0-0.x86_64 kubelet-1.20.4-0.x86_64 kubeadm-1.20.4-0.x86_64
-->重启kubelet
命令:
systemctl restart kubelet
systemctl status kubelet
-->查看kubelet日志(正常)
命令:
journalctl -xefu kubelet
-->节点验证
命令:
rpm -qa | grep kube
kubectl -n kube-system get node
kubectl -n kube-system get pods
[root@centos79-3 ~]# rpm -qa | grep kube
kubectl-1.20.4-0.x86_64
kubernetes-cni-1.2.0-0.x86_64
kubelet-1.20.4-0.x86_64
kubeadm-1.20.4-0.x86_64
[root@centos79-3 ~]# kubectl -n kube-system get node
NAME STATUS ROLES AGE VERSION
cent79-2 Ready <none> 2d22h v1.20.4
centos79 Ready <none> 4h23m v1.20.4
centos79-3 Ready control-plane,master 2d22h v1.20.4
[root@centos79-3 ~]# kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-7854b85cf7-hn2zk 1/1 Running 10 2d18h
calico-node-f2s8w 1/1 Running 7 2d18h
calico-node-pbjg4 1/1 Running 2 4h23m
calico-node-v8cqp 1/1 Running 0 92m
coredns-7f89b7bc75-plrq9 1/1 Running 10 2d22h
coredns-7f89b7bc75-xdx8j 1/1 Running 10 2d22h
etcd-centos79-3 1/1 Running 12 2d22h
kube-apiserver-centos79-3 1/1 Running 19 2d22h
kube-controller-manager-centos79-3 1/1 Running 12 2d22h
kube-proxy-5srcv 1/1 Running 0 93m
kube-proxy-bpg48 1/1 Running 3 25h
kube-proxy-kf48c 1/1 Running 3 25h
kube-scheduler-centos79-3 1/1 Running 13 2d22h
[root@centos79-3 ~]#
至此, k8s kubelet日志报..." err="failed to parse kubelet flag: unknown flag: --network-plugin"问题完成解决