kube-api log:authentication handshake failed: x509: certificate has expired or is not yet valid

问题描述：

[root@master .kube]# kubectl get nodes
The connection to the server 192.168.122.2:6443 was refused - did you specify the right host or port?

[root@master ~]# docker ps  |head -1 ; docker ps |grep api
CONTAINER ID   IMAGE                                                           COMMAND                  CREATED          STATUS          PORTS     NAMES
9425402ec49a   838d692cbe28                                                    "kube-apiserver --ad…"   16 seconds ago   Up 16 seconds             k8s_kube-apiserver_kube-apiserver-master_kube-system_29f37e829364bd5dd2a022f9cde4d40e_139
9c00a266f9b2   registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5   "/pause"                 5 minutes ago    Up 5 minutes              k8s_POD_kube-apiserver-master_kube-system_29f37e829364bd5dd2a022f9cde4d40e_37
[root@master ~]# 

[root@master ~]# docker logs 9425402ec49a
I0925 17:40:21.641822       1 server.go:553] external host was not specified, using 192.168.122.2
I0925 17:40:21.642866       1 server.go:161] Version: v1.22.0
I0925 17:40:22.208542       1 shared_informer.go:240] Waiting for caches to sync for node_authorizer
I0925 17:40:22.210936       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0925 17:40:22.211062       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0925 17:40:22.213863       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0925 17:40:22.213897       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0925 17:40:22.225185       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:22Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:23.205264       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:23Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:23.230706       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:23Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:24.211317       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:24Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:24.706423       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:24Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:25.543315       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:25Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:27.504461       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:27Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:27.615949       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:27Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:31.099569       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:31Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:32.038615       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:32Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:36.806572       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:36Z is after 2022-08-14T04:09:37Z". Reconnecting...
W0925 17:40:37.627922       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid: current time 2022-09-25T17:40:37Z is after 2022-08-14T04:09:37Z". Reconnecting...
Error: context deadline exceeded

解决方法： 

If you are on K8s 1.17.9 or above, the following worked:

kubeadm alpha certs check-expiration; kubeadm alpha certs renew all
Recent versions do not require the "alpha tag" anymore. For these, just use this:

kubeadm certs check-expiration; kubeadm certs renew all
 

[root@master ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 14, 2022 04:09 UTC   <invalid>                               no      
apiserver                  Aug 14, 2022 04:09 UTC   <invalid>       ca                      no      
apiserver-etcd-client      Aug 14, 2022 04:09 UTC   <invalid>       etcd-ca                 no      
apiserver-kubelet-client   Aug 14, 2022 04:09 UTC   <invalid>       ca                      no      
controller-manager.conf    Aug 14, 2022 04:09 UTC   <invalid>                               no      
etcd-healthcheck-client    Aug 14, 2022 04:09 UTC   <invalid>       etcd-ca                 no      
etcd-peer                  Aug 14, 2022 04:09 UTC   <invalid>       etcd-ca                 no      
etcd-server                Aug 14, 2022 04:09 UTC   <invalid>       etcd-ca                 no      
front-proxy-client         Aug 14, 2022 04:09 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf             Aug 14, 2022 04:09 UTC   <invalid>                               no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Aug 12, 2031 04:09 UTC   8y              no      
etcd-ca                 Aug 12, 2031 04:09 UTC   8y              no      
front-proxy-ca          Aug 12, 2031 04:09 UTC   8y              no      

[root@master ~]# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
[root@master ~]# 

then copy /etc/kubernetes/admin.conf to your ~/.kube/config 

[root@master ~]# cp /etc/kubernetes/admin.conf ~/.kube/config
cp: overwrite ‘/root/.kube/config’? y
[root@master ~]# 

In order for the cluster to actually reload the keys, after you received the following message:

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
reload the relevant services with:

[root@master ~]# kubectl -n kube-system delete pod -l 'component=kube-apiserver'
pod "kube-apiserver-master" deleted
[root@master ~]# kubectl -n kube-system delete pod -l 'component=kube-controller-manager'
pod "kube-controller-manager-master" deleted
[root@master ~]# kubectl -n kube-system delete pod -l 'component=kube-scheduler'
pod "kube-scheduler-master" deleted
[root@master ~]# kubectl -n kube-system delete pod -l 'component=etcd'
pod "etcd-master" deleted
[root@master ~]# 

Issue resolved:

[root@master ~]# kubectl get nodes
NAME     STATUS   ROLES                  AGE    VERSION
master   Ready    control-plane,master   420d   v1.22.0
node1    Ready    <none>                 412d   v1.22.0
node2    Ready    <none>                 412d   v1.22.0
[root@master ~]# 




[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE              NAME                                                              READY   STATUS             RESTARTS          AGE
default                nfs-client-provisioner-6cf7cdc4fd-98sqw                           1/1     Running            1 (60d ago)       76d
default                postgressqldb-78fbf89b68-bq8tp                                    0/1     ImagePullBackOff   199 (63d ago)     76d
default                sklmapp-875588558-g9lmj                                           0/1     CrashLoopBackOff   193 (9m17s ago)   76d
ibm-common-services    ibm-licensing-operator-85554b699d-t67c2                           1/1     Running            1 (60d ago)       72d
ibm-common-services    ibm-licensing-service-instance-6c56c44d78-d296g                   1/1     Running            1 (60d ago)       72d
kube-system            calico-kube-controllers-58497c65d5-xhfsf                          1/1     Running            396 (60d ago)     407d
kube-system            calico-node-cwqv4                                                 1/1     Running            52 (60d ago)      407d
kube-system            calico-node-fngr6                                                 1/1     Running            49 (60d ago)      400d
kube-system            calico-node-tv2zq                                                 1/1     Running            59 (60d ago)      400d