版本信息
该方案在kubernetes 1.12.5下验证有效,其他版本可能会略有不同,但基本原理相似。
故障现象
使用kubeadm部署的集群,默认根证书有效期是10年,其余证书的有效期为1年。当证书到期之后,k8s各核心组件之间的交互和协作会因为证书验证失效到时失败,从而导致k8s集群不可用。
故障排查和验证
故障排查
查看k8s各组件的日志会发现ssl认证错误信息。
故障验证
通过openssl x509 -in [证书名称] -noout -text
命令能够查看证书的相关信息,从而确定是否证书到期:
$ cd /etc/kubernetes/pki
$ openssl x509 -in apiserver-kubelet-client.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 5363063852586150726 (0x4a6d6e28c122ab46)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Dec 16 12:31:51 2019 GMT
Not After : Dec 15 12:31:52 2020 GMT
Subject: O=system:masters, CN=kube-apiserver-kubelet-client
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:df:72:40:f1:6f:0d:02:7d:d6:ce:d4:55:0a:e2:
aa:12:e9:dc:2d:7e:b7:a7:5f:42:e5:9b:e2:3d:dd:
b8:ec:7c:d8:b9:36:33:5b:07:76:ec:e0:f4:75:34:
8a:94:0d:21:f4:3d:66:2f:9e:95:58:62:d3:d7:ee:
25:f0:c3:ea:fb:4c:30:6e:18:ef:78:40:61:6b:3b:
55:12:f7:72:42:b6:87:c0:85:2f:03:d2:b9:a9:72:
3e:12:ae:db:a0:e4:37:13:18:35:ad:7f:41:ab:7a:
3f:55:a9:68:9e:24:2a:69:db:9f:1e:6f:10:9f:9f:
d0:6b:76:83:28:02:36:29:2b:41:53:87:63:36:c1:
79:aa:58:35:6a:3f:46:9f:b7:7d:58:cc:1a:c2:d8:
85:de:5b:81:07:2d:08:cd:ef:d0:08:e2:b2:05:2d:
9c:27:cb:4e:e4:55:e6:2b:26:86:4b:11:a3:d1:27:
53:f4:b9:75:5d:1d:fc:ee:c5:1d:07:6a:27:e3:a1:
0f:ae:b7:2b:14:01:b4:ff:46:0b:d0:4b:4e:fa:5b:
94:93:66:84:2f:49:ed:9c:23:65:bf:09:ee:69:c8:
6a:66:be:54:09:46:d0:5c:7f:a4:bd:9a:2c:5d:66:
b0:c6:ba:3c:f5:b7:8a:3c:b1:ba:3f:9c:b3:98:44:
41:7d
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
Signature Algorithm: sha256WithRSAEncryption
97:fa:aa:29:13:16:43:3c:66:f4:3e:c9:6c:18:9b:86:fb:ef:
fb:12:2a:b8:64:cd:f6:44:b4:49:70:5b:b7:f1:a6:df:68:55:
61:24:18:2a:ef:5e:19:6e:0e:c7:48:1f:ff:cc:bc:6e:d0:5e:
b9:3d:59:46:a2:0b:0b:ce:6d:ca:ce:58:8e:b2:7c:fe:0d:70:
14:11:20:1c:e8:9b:0c:e4:fe:f3:81:c3:d4:e5:65:16:f3:d8:
a1:3e:c1:c5:e9:31:e9:6e:7c:7e:29:69:52:06:cc:f2:4c:ad:
bd:52:09:55:3f:a9:ce:fb:84:df:d4:b4:04:de:dd:01:35:4f:
6e:25:d2:d7:12:5b:34:05:8a:f0:f2:19:34:16:9f:f3:96:c3:
24:ab:e0:be:01:70:fa:bc:4f:f2:31:43:bf:db:f8:de:6a:6d:
0d:78:b1:8d:16:3e:53:6d:92:17:41:e1:c3:6a:92:e3:83:f9:
3e:82:0e:56:7f:6d:cb:b8:aa:e2:bf:77:69:8f:13:a2:67:e4:
df:e3:21:a6:12:f8:a4:bc:02:c7:70:0b:67:07:5b:61:97:f6:
fa:74:0a:36:9c:46:5a:08:c3:70:2a:64:9b:66:ff:ec:fe:f2:
80:84:06:fa:8f:c4:45:de:28:45:34:1b:f3:18:8f:db:5a:b8:
59:a5:9f:76
数据备份
etcd数据备份
通常,etcd数据存储在/var/lib/etcd
目录下,如果不是,请核实etcd数据存储路径,然后备份。
$ cp -r /var/lib/etcd /var/lib/etcd-bak
证书备份
$ mkdir -p /etc/kubernetes/pki-bak
$ mv /etc/kubernetes/pki/*.crt /etc/kubernetes/pki-bak
$ mv /etc/kubernetes/pki/*.key /etc/kubernetes/pki-bak
conf文件备份
$ mkdir -p /etc/kubernetes/conf
$ mv /etc/kubernetes/*.conf /etc/kubernetes/conf
生成新证书
$ kubeadm alpha phase certs apiserver --apiserver-advertise-address ${MASTER_API_SERVER_IP}
$ kubeadm alpha phase certs apiserver-kubelet-client
$ kubeadm alpha phase certs front-proxy-client
生成新配置文件
$ kubeadm alpha phase kubeconfig all --apiserver-advertise-address ${MASTER_API_SERVER_IP}
更新admin配置文件
$ cp /etc/kubernetes/admin.conf /root/.kube/conf
重建服务
$ cd /etc/kubernetes/manifests
$ mv *.yaml ..
$ mv ../*.yaml .
验证
通过如下命令查看node的健康状态,如果证书无误,所有节点将变为Ready
状态。
$ kubectl get node