情况说明
原始集群中使用flannel作为网络插件,后来想在集群中使用calico网络插件建立网络规则,但在安装的过程中因为一些失误操作(没有读懂官方的安装说明),导致集群宕机,coredns始终处于ContainerCreating状态。
报错信息:
首先需要说明一下,coredns的状态和flannel有着必然的联系,所以首先使用describe查看pods的错误信息。
[root@master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5bfd685c78-mmjxc 0/1 ContainerCreating 0 9s
coredns-5bfd685c78-zmmpv 0/1 ContainerCreating 0 39s
etcd-master 1/1 Running 5 47d
kube-apiserver-master 1/1 Running 5 47d
kube-controller-manager-master 1/1 Running 5 47d
kube-flannel-ds-8vzsv 1/1 Running 1 15m
kube-flannel-ds-zbqt9 1/1 Running 1 15m
kube-flannel-ds-zwxrh 1/1 Running 1 15m
kube-proxy-6r25s 1/1 Running 5 47d
kube-proxy-m8gxx 1/1 Running 5 47d
kube-proxy-s2jb8 1/1 Running 5 47d
kube-scheduler-master 1/1 Running 5 47d
使用describe查看错误信息。
[root@master ~]# kubectl describe pods -n kube-system coredns-5bfd685c78-mmjxc
Name: coredns-5bfd685c78-mmjxc
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: node2/192.168.10.220
Start Time: Tue, 14 Jul 2020 22:11:33 +0800
Labels: k8s-app=kube-dns
pod-template-hash=5bfd685c78
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/coredns-5bfd685c78
Containers:
coredns:
Container ID:
Image: k8s.gcr.io/coredns:1.3.1
Image ID:
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-vcfp8 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-vcfp8:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-vcfp8
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 25s default-scheduler Successfully assigned kube-system/coredns-5bfd685c78-mmjxc to node2
Warning FailedCreatePodSandBox 24s kubelet, node2 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "071c8fc6acb87838fd4ee341479a0769a97401c481a93b5b54f8812ba6fa0ed4" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 23s kubelet, node2 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e35ce83b7c4f18d1aa8c2a57cea6ade03b33ad98041d1b5c0d218ddfc300f23e" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 22s kubelet, node2 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e043cc0de7e693fa36d8115ae07b44560ab1883363127fca15becc36b7976ecc" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 21s kubelet, node2 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d128ed26899ab9cf2bf9f62784a3bb723b63141cd3fff82f860325ad3433c322" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 20s kubelet, node2 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "045f258154329ba4e46192a01df441cf04c395a3599fd68ebd5a5bf7c648fcd1" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
Warning FailedCreatePodSandBox 19s kubelet, node2 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dc776f01abbbf1a219f8014c377f0359db90c4ec74cfe8baa993ff63a0b805f7" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
可以看出node节点中的网络不通畅,
解决方法
卸载master、node节点上所有关于calico的安装信息。
l连接Node节点,使用以下命令删除node节点上关于calico的配置信息,并重启kubelet服务
rm -rf /etc/cni/net.d/*
rm -rf /var/lib/cni/calico
systemctl restart kubelet