目录
一.排错思路
1.查看报错
# 节点状态 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready control-plane,master 2d17h v1.23.2 vm-0-114-centos Ready <none> 23h v1.23.2 vm-0-142-centos NoReady <none> 23s v1.23.2 # 组件状态 ~]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-557689b88f-g9pwl 1/1 Running 1 (102m ago) 2d18h kube-system coredns-557689b88f-rj84r 1/1 Running 1 (102m ago) 2d18h kube-system etcd-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-apiserver-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-controller-manager-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-flannel-ds-jxwf5 0/1 pending 0 81m kube-system kube-flannel-ds-qk97b 1/1 Running 1 (102m ago) 11h kube-system kube-flannel-ds-qzhqt 1/1 Running 0 11h kube-system kube-proxy-flzrz 1/1 Running 2 (102m ago) 2d18h kube-system kube-proxy-sv5n8 1/1 Running 0 81m kube-system kube-proxy-tkctb 1/1 Running 0 24h kube-system kube-scheduler-master 1/1 Running 4 (102m ago) 2d18h
注;可以再精确一点kubectl get pods -A -o wide查看是哪节点报错
2.查看系统环境版本
1)系统版本
Master节点
~]# cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"
node节点
~]# cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"
2)K8S版本
master节点
~]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:34:34Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
node节点
~]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:34:34Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
3)Docker版本
master节点
~]# docker version Client: Docker Engine - Community Version: 20.10.12 API version: 1.41 Go version: go1.16.12 Git commit: e91ed57 Built: Mon Dec 13 11:45:41 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.12 API version: 1.41 (minimum version 1.12) Go version: go1.16.12 Git commit: 459d0df Built: Mon Dec 13 11:44:05 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.12 GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2 docker-init: Version: 0.19.0 GitCommit: de40ad0
node节点
~]# docker version Client: Docker Engine - Community Version: 20.10.12 API version: 1.41 Go version: go1.16.12 Git commit: e91ed57 Built: Mon Dec 13 11:45:41 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.12 API version: 1.41 (minimum version 1.12) Go version: go1.16.12 Git commit: 459d0df Built: Mon Dec 13 11:44:05 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.12 GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2 docker-init: Version: 0.19.0 GitCommit: de40ad0
4)查看hosts文件配置
master节点
~]# cat /etc/hosts ..... 172.17.0.51 master 172.17.0.114 node1 172.17.0.142 node2 172.17.0.98 registry
node节点
~]# cat /etc/hosts ..... 172.17.0.51 master 172.17.0.114 node1 172.17.0.142 node2 172.17.0.98 registry
5) 主机通讯检测
master节点
~]# ping node2 -c 5 PING node2 (172.17.0.142) 56(84) bytes of data. 64 bytes from node2 (172.17.0.142): icmp_seq=1 ttl=64 time=0.231 ms 64 bytes from node2 (172.17.0.142): icmp_seq=2 ttl=64 time=0.263 ms 64 bytes from node2 (172.17.0.142): icmp_seq=3 ttl=64 time=0.188 ms 64 bytes from node2 (172.17.0.142): icmp_seq=4 ttl=64 time=0.220 ms 64 bytes from node2 (172.17.0.142): icmp_seq=5 ttl=64 time=0.182 ms --- node2 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.182/0.216/0.263/0.034 ms
node节点
~]# ping master -c 5 PING master (172.17.0.51) 56(84) bytes of data. 64 bytes from master (172.17.0.51): icmp_seq=1 ttl=64 time=0.188 ms 64 bytes from master (172.17.0.51): icmp_seq=2 ttl=64 time=0.212 ms 64 bytes from master (172.17.0.51): icmp_seq=3 ttl=64 time=0.216 ms 64 bytes from master (172.17.0.51): icmp_seq=4 ttl=64 time=0.222 ms 64 bytes from master (172.17.0.51): icmp_seq=5 ttl=64 time=0.215 ms --- master ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.188/0.210/0.222/0.019 ms
6)配置网桥监控主机流量,主机路由转发
master
~]# sysctl --system ..... * Applying /etc/sysctl.d/k8s.conf ... net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1
node
~]# sysctl --system ..... * Applying /etc/sysctl.d/k8s.conf ... net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1
7)指定私有镜像仓库文件
master
~]# cat /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "registry-mirrors": ["https://hub-mirror.c.163.com"], "insecure-registries":["172.17.0.98:5000", "registry:5000"] }
node
~]# cat /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "registry-mirrors": ["https://hub-mirror.c.163.com"], "insecure-registries":["172.17.0.98:5000", "registry:5000"] }
8)查看节点容器运行状态
node1
~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 172.17.0.98:5000/kube-apiserver v1.23.2 8a0228dd6a68 5 days ago 135MB 172.17.0.98:5000/kube-controller-manager v1.23.2 4783639ba7e0 5 days ago 125MB 172.17.0.98:5000/kube-scheduler v1.23.2 6114d758d6d1 5 days ago 53.5MB 172.17.0.98:5000/kube-proxy v1.23.2 d922ca3da64b 5 days ago 112MB 172.17.0.98:5000/etcd 3.5.1-0 25f8c7f3da61 2 months ago 293MB 172.17.0.98:5000/coredns v1.8.6 a4ca41631cc7 3 months ago 46.8MB 172.17.0.98:5000/pause 3.6 6270bb605e12 5 months ago 683kB
node2
~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c67993ac008d 404fc3ab6749 "/opt/bin/flanneld -…" 2 hours ago Up 2 hours k8s_kube-flannel_kube-flannel-ds-jxwf5_kube-system_e8b3608e-9059-49a7-b5fd-d6244e3bd06c_0 79925c61f378 404fc3ab6749 "cp -f /etc/kube-fla…" 2 hours ago Exited (0) 2 hours ago k8s_install-cni_kube-flannel-ds-jxwf5_kube-system_e8b3608e-9059-49a7-b5fd-d6244e3bd06c_0 13251ae462b1 cd5235cd7dc2 "cp -f /flannel /opt…" 2 hours ago Exited (0) 2 hours ago k8s_install-cni-plugin_kube-flannel-ds-jxwf5_kube-system_e8b3608e-9059-49a7-b5fd-d6244e3bd06c_0 1bb3845bfa4a d922ca3da64b "/usr/local/bin/kube…" 2 hours ago Up 2 hours k8s_kube-proxy_kube-proxy-sv5n8_kube-system_df8c76f4-89ef-41e6-83c0-a678ce3fd986_0 a024c30cd119 172.17.0.98:5000/pause:3.6 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-proxy-sv5n8_kube-system_df8c76f4-89ef-41e6-83c0-a678ce3fd986_0 c8a4190d3582 172.17.0.98:5000/pause:3.6 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-flannel-ds-jxwf5_kube-system_e8b3608e-9059-49a7-b5fd-d6244e3bd06c_0 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 172.17.0.98:5000/kube-apiserver v1.23.2 8a0228dd6a68 5 days ago 135MB 172.17.0.98:5000/kube-controller-manager v1.23.2 4783639ba7e0 5 days ago 125MB 172.17.0.98:5000/kube-scheduler v1.23.2 6114d758d6d1 5 days ago 53.5MB 172.17.0.98:5000/kube-proxy v1.23.2 d922ca3da64b 5 days ago 112MB rancher/mirrored-flannelcni-flannel v0.16.1 404fc3ab6749 2 weeks ago 69.9MB 172.17.0.98:5000/etcd 3.5.1-0 25f8c7f3da61 2 months ago 293MB rancher/mirrored-flannelcni-flannel-cni-plugin v1.0.0 cd5235cd7dc2 2 months ago 9.03MB 172.17.0.98:5000/coredns v1.8.6 a4ca41631cc7 3 months ago 46.8MB 172.17.0.98:5000/pause 3.6 6270bb605e12 5 months ago 683kB
注:此时就可以查看到是容器镜像没有导致容器未启动
二.问题解决
1.删除node2节点及配置
1)删除node2节点
# 先查看节点名称 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready control-plane,master 2d19h v1.23.2 vm-0-114-centos Ready <none> 25h v1.23.2 vm-0-142-centos NoReady <none> 114m v1.23.2 # 删除节点 ~]# kubectl delete node vm-0-142-centos node "vm-0-142-centos" deleted # 查看删除结果 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready control-plane,master 2d17h v1.23.2 vm-0-114-centos Ready <none> 22h v1.23.2
2)删除所有容器(node2)
~]# docker ps -qa | xargs docker rm -f
3)删除旧的配置文件
~]# rm -f /etc/kubernetes/kubelet.conf
4)删除旧的ca文件
~]# rm -f /etc/kubernetes/pki/ca.crt
5)重启k8s及docker服务(node2)
~]# systemctl restart docker kubelet
6)手动导入镜像
~]# docker pull rancher/mirrored-flannelcni-flannel:v0.16.1 ~]# docker pull rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0
2.重新注册,加入集群
1)生成注册命令(master)
~]# kubeadm token create --print-join-command kubeadm join 172.17.0.51:6443 --token 3mucph.ml5h85umn1flgmir --discovery-token-ca-cert-hash sha256:72db7edc40a9581a8e9f96853a4e3fb03
2)加入集群(node)
~]# kubeadm join 172.17.0.51:6443 --token 3mucph.ml5h85umn1flgmir --discovery-token-ca-cert-hash sha256:72db7edc40a9581a8e9f96853a4e3fb03 [preflight] Running pre-flight checks [WARNING Hostname]: hostname "vm-0-142-centos" could not be reached [WARNING Hostname]: hostname "vm-0-142-centos": lookup vm-0-142-centos on 183.60.83.19:53: no such host [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
3) 查看状态(master)
# 查看node节点状态 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready control-plane,master 2d17h v1.23.2 vm-0-114-centos Ready <none> 23h v1.23.2 vm-0-142-centos Ready <none> 25s v1.23.2 # 查看pod状态 ~]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-557689b88f-g9pwl 1/1 Running 1 (102m ago) 2d18h kube-system coredns-557689b88f-rj84r 1/1 Running 1 (102m ago) 2d18h kube-system etcd-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-apiserver-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-controller-manager-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-flannel-ds-jxwf5 1/1 Running 0 81m kube-system kube-flannel-ds-qk97b 1/1 Running 1 (102m ago) 11h kube-system kube-flannel-ds-qzhqt 1/1 Running 0 11h kube-system kube-proxy-flzrz 1/1 Running 2 (102m ago) 2d18h kube-system kube-proxy-sv5n8 1/1 Running 0 81m kube-system kube-proxy-tkctb 1/1 Running 0 24h kube-system kube-scheduler-master 1/1 Running 4 (102m ago) 2d18h
至此,问题解决!所以导致这个问题的主要原因就是前期导入镜像导致 ,但是此方法过与复杂,
3.解决另一种方法
注:这里就不需要删除节点了
## 因为我们只是镜像没有,所以只需以下几步 # 第一步到node节点下传镜像 ~]# docker pull rancher/mirrored-flannelcni-flannel:v0.16.1 ~]# docker pull rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0 # 第二部重启flannel ~]# kubectl delete -f kube-flannel.yml ~]# kubectl apply -f kube-flannel.yml # 查看 # 查看node节点状态 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready control-plane,master 2d17h v1.23.2 vm-0-114-centos Ready <none> 23h v1.23.2 vm-0-142-centos Ready <none> 25s v1.23.2 # 查看pod状态 ~]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-557689b88f-g9pwl 1/1 Running 1 (102m ago) 2d18h kube-system coredns-557689b88f-rj84r 1/1 Running 1 (102m ago) 2d18h kube-system etcd-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-apiserver-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-controller-manager-master 1/1 Running 3 (102m ago) 2d18h kube-system kube-flannel-ds-jxwf5 1/1 Running 0 81m kube-system kube-flannel-ds-qk97b 1/1 Running 1 (102m ago) 11h kube-system kube-flannel-ds-qzhqt 1/1 Running 0 11h kube-system kube-proxy-flzrz 1/1 Running 2 (102m ago) 2d18h kube-system kube-proxy-sv5n8 1/1 Running 0 81m kube-system kube-proxy-tkctb 1/1 Running 0 24h kube-system kube-scheduler-master 1/1 Running 4 (102m ago) 2d18h