kubernetes集群一个master两个node证书过期解决

1.集群表现现象：

1.kubelet无法启动，并报如下错误：

[root@k8s01 ~]# journalctl -u kubelet.service  -f
-- Logs begin at Tue 2023-10-10 10:07:58 CST. --
Nov 10 15:30:31 k8s01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Nov 10 15:30:31 k8s01 kubelet[25591]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 10 15:30:31 k8s01 kubelet[25591]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 10 15:30:31 k8s01 kubelet[25591]: I1110 15:30:31.153621   25591 server.go:440] "Kubelet version" kubeletVersion="v1.21.2"
Nov 10 15:30:31 k8s01 kubelet[25591]: I1110 15:30:31.154104   25591 server.go:851] "Client rotation is on, will bootstrap in background"
Nov 10 15:30:31 k8s01 kubelet[25591]: E1110 15:30:31.157009   25591 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2024-10-08 06:25:08 +0000 UTC
Nov 10 15:30:31 k8s01 kubelet[25591]: E1110 15:30:31.157090   25591 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
Nov 10 15:30:31 k8s01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 10 15:30:31 k8s01 systemd[1]: Unit kubelet.service entered failed state.
Nov 10 15:30:31 k8s01 systemd[1]: kubelet.service failed.
Nov 10 15:30:41 k8s01 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Nov 10 15:30:41 k8s01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Nov 10 15:30:41 k8s01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Nov 10 15:30:41 k8s01 kubelet[25676]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 10 15:30:41 k8s01 kubelet[25676]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 10 15:30:41 k8s01 kubelet[25676]: I1110 15:30:41.404211   25676 server.go:440] "Kubelet version" kubeletVersion="v1.21.2"
Nov 10 15:30:41 k8s01 kubelet[25676]: I1110 15:30:41.404751   25676 server.go:851] "Client rotation is on, will bootstrap in background"
Nov 10 15:30:41 k8s01 kubelet[25676]: E1110 15:30:41.407590   25676 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2024-10-08 06:25:08 +0000 UTC
Nov 10 15:30:41 k8s01 kubelet[25676]: E1110 15:30:41.407670   25676 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
Nov 10 15:30:41 k8s01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 10 15:30:41 k8s01 systemd[1]: Unit kubelet.service entered failed state.
Nov 10 15:30:41 k8s01 systemd[1]: kubelet.service failed.

【分析】：

• failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

在前边我们就可以看到这个报错，如果你对k8s的认证比较了解的话，就会知道bootstrap-kubelet.conf是k8s API的引导令牌（Bootstrap Tokens）认证相关的文件。该机制根据证书生成token，然后将信息写字这个文件里。

• kubectl命令报错如下：

[root@k8s01 ~]# kubectl get nodes
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2024-11-10T15:30:55+08:00 is after 2024-10-08T06:25:06Z

【分析】：

这里我们可以定位到证书问题，证书过期。

2.解决集群问题

• 备份并重新生成证书

[root@k8s01 ~]# cd /etc/kubernetes/pki/
[root@k8s01 pki]# mkdir backup
[root@k8s01 pki]# mv  apiserver.crt apiserver-etcd-client.key apiserver-kubelet-client.crt front-proxy-ca.crt front-proxy-client.crt front-proxy-client.key front-proxy-ca.key apiserver-kubelet-client.key apiserver.key apiserver-etcd-client.crt backup
[root@k8s01 pki]# kubeadm init phase certs all

• 备份并重新生成配置文件

[root@k8s01 pki]# cd /etc/kubernetes/
[root@k8s01 kubernetes]# mkdir backup
[root@k8s01 kubernetes]# mv admin.conf controller-manager.conf kubelet.conf scheduler.conf  backup
[root@k8s01 kubernetes]# kubeadm  init phase kubeconfig all

• 拷贝用户权限文件

[root@k8s01 kubernetes]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

• 查看证书

[root@k8s01 ~]# kubeadm  certs check-expiration

• 启动kubelet

[root@k8s01 ~]# systemctl restart kubelet
[root@k8s01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Tue 2023-10-10 13:53:03 CST; 6s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 5111 (kubelet)
    Tasks: 14
   Memory: 29.0M
   CGroup: /system.slice/kubelet.service

• 处理工作节点证书

删除工作节点：
[root@k8s01 ~]# kubectl delete nodes k8s02
[root@k8s01 ~]# kubectl delete nodes k8s03
生成加入节点命令：
[root@k8s01 ~]# kubeadm token create --print-join-command
kubeadm join 172.11.0.74:6443 --token msbk7j.8ynbop6j6irn9qw7 --discovery-token-ca-cert-hash sha256:d5bd944fa55cd38f8a2515958e9523f85dbae93e3e91e7b50a15cc9b33d196f0
把工作节点重新加入节点：
[root@k8s02 ~]# kubeadm reset -f
[root@k8s02 ~]# kubeadm join 172.11.0.74:6443 --token msbk7j.8ynbop6j6irn9qw7 --discovery-token-ca-cert-hash sha256:d5bd944fa55cd38f8a2515958e9523f85dbae93e3e91e7b50a15cc9b33d196f0
[root@k8s03 ~]# kubeadm reset -f
[root@k8s03 ~]# kubeadm join 172.11.0.74:6443 --token msbk7j.8ynbop6j6irn9qw7 --discovery-token-ca-cert-hash sha256:d5bd944fa55cd38f8a2515958e9523f85dbae93e3e91e7b50a15cc9b33d196f0