kubeadm 重新生成证书

Kubernetes 各个组件需要与 api-server 进行通信，通信使用的证书都存放在 /etc/kubernetes/pki 路径下，kubeadm 生成的证书默认有效为 1 年，因此需要定时更新证书，否则证书到期会导致整个集群不可用。（当然如果你想，你也可以随时重新生成证书）

一、检查证书是否过期。

可以通过下面两种方式检查 Kubernetes 的证书是否过期。

1. kubeadm 命令查看

可以通过 kubeadm certs check-expiration 命令查看相关证书是否过期。

$ kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1121 17:11:46.728757   24104 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.*.*.10]; the provided value is: [169.*.*.10]

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Nov 20, 2024 07:52 UTC   364d                                    no
apiserver                  Nov 20, 2024 06:58 UTC   364d            ca                      no
apiserver-kubelet-client   Nov 20, 2024 06:58 UTC   364d            ca                      no
controller-manager.conf    Nov 20, 2024 07:52 UTC   364d                                    no
front-proxy-client         Nov 20, 2024 06:58 UTC   364d            front-proxy-ca          no
scheduler.conf             Nov 20, 2024 07:52 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 17, 2033 10:57 UTC   9y              no
front-proxy-ca          Nov 17, 2033 10:12 UTC   9y              no

1. openssl 命令查看

可以通过 openssl 查看对应证书是否过期。

$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Oct 20 10:57:32 2023 GMT
            Not After : Nov 20 06:58:35 2024 GMT

二. 自动更新证书

Kubenetes 在升级控制面板相关组件时会主动更新证书，因此如果保证 Kubernetes 能够定期（一年以内）升级的话，证书会自动更新。

三. 手动更新证书

1. 证书备份

$ cp -rp /etc/kubernetes /etc/kubernetes.bak

1. 删除旧的证书

将 /etc/kubernetes/pki 下要重新生成的证书删除

$ rm -rf /etc/kubernetes/pki/apiserver.key

1. 重新生成证书

$ kubeadm certs renew  -h
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm certs renew [flags]
  kubeadm certs renew [command]

Available Commands:
  admin.conf               Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
  all                      Renew all available certificates
  apiserver                Renew the certificate for serving the Kubernetes API
  apiserver-etcd-client    Renew the certificate the apiserver uses to access etcd
  apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
  controller-manager.conf  Renew the certificate embedded in the kubeconfig file for the controller manager to use
  etcd-healthcheck-client  Renew the certificate for liveness probes to healthcheck etcd
  etcd-peer                Renew the certificate for etcd nodes to communicate with each other
  etcd-server              Renew the certificate for serving etcd
  front-proxy-client       Renew the certificate for the front proxy client
  scheduler.conf           Renew the certificate embedded in the kubeconfig file for the scheduler manager to use

Flags:
  -h, --help   help for renew

Global Flags:
      --add-dir-header           If true, adds the file directory to the header of the log messages
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --one-output               If true, only write logs to their native severity level (vs also writing to each lower severity level)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm certs renew [command] --help" for more information about a command.

重新生成所有证书

$ kubeadm certs renew all --config /etc/kubernetes/kubeadm-config.yaml
W1121 17:22:07.996499   38007 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.*.0.*]; the provided value is: [*.*.*.10]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
$ ll -tr
总用量 64
-rw------- 1 root root 1675 11月 20 18:12 front-proxy-ca.key
-rw-r--r-- 1 root root 1078 11月 20 18:12 front-proxy-ca.crt
-rw------- 1 root root  451 11月 20 18:12 sa.pub
-rw------- 1 root root 1675 11月 20 18:12 sa.key
-rw-r--r-- 1 root root 1066 11月 21 09:05 ca.crt1
-rw------- 1 root root 1679 11月 21 09:05 ca.key1
-rw-r--r-- 1 root root 1066 11月 21 09:06 ca.crt
-rw------- 1 root root 1675 11月 21 09:07 ca.key
-rw------- 1 root root 1679 11月 21 09:08 apiserver.key1
-rw-r--r-- 1 root root  899 11月 21 16:13 apiserver.csr
-rw------- 1 root root 1679 11月 21 17:22 apiserver.key
-rw-r--r-- 1 root root 1472 11月 21 17:22 apiserver.crt
-rw------- 1 root root 1675 11月 21 17:22 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1164 11月 21 17:22 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 11月 21 17:22 front-proxy-client.key
-rw-r--r-- 1 root root 1119 11月 21 17:22 front-proxy-client.crt

会发现更新了apiserver.key，apiserver.crt等一系列文件

重新生成某个组件的证书

kubeadm certs renew apiserver

1. 重新生成配置文件

删除旧的配置

$ rm -rf /etc/kubernetes/*.conf

生成新的配置

主要通过 kubeadm init phase kubeconfig 命令执行：

$ kubeadm init phase kubeconfig
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm init phase kubeconfig [flags]
  kubeadm init phase kubeconfig [command]

Available Commands:
  admin              Generate a kubeconfig file for the admin to use and for kubeadm itself
  all                Generate all kubeconfig files
  controller-manager Generate a kubeconfig file for the controller manager to use
  kubelet            Generate a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes
  scheduler          Generate a kubeconfig file for the scheduler to use

Flags:
  -h, --help   help for kubeconfig

Global Flags:
      --add-dir-header           If true, adds the file directory to the header of the log messages
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --one-output               If true, only write logs to their native severity level (vs also writing to each lower severity level)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm init phase kubeconfig [command] --help" for more information about a command.

重新生成所有配置

$ kubeadm init phase kubeconfig all
I1121 17:28:57.934884   46841 version.go:254] remote version is much newer: v1.28.4; falling back to: stable-1.21
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file

重新生成单个配置文件

// 重新生成 admin 配置文件
kubeadm init phase kubeconfig admin
// 重新生成 kubelet 配置文件
kubeadm init phase kubeconfig kubelet

1. 重启服务

完成证书和配置文件的更新后，需要进行一系列后续操作保证更新生效，主要包括重启 kubelet、api-server、schedule等更新管理配置。

重启kube-apiserver、kube-controller-manager和kube-scheduler

将manifests下面yaml文件移除，container会自动删除，再移入会自动启动，这样就完成了

$ pwd
/etc/kubernetes/manifests
$ ls
kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
$ mv manifests/* /tmp/
$ mv /tmp/kube-* /etc/kubernetes/manifests/
$ docker ps | grep kube
a0b5a5ffc637   7b2ac941d4c3             "kube-apiserver --ad…"   6 seconds ago       Up 5 seconds                 k8s_kube-apiserver_kube-apiserver-ecs-89475272-001_kube-system_98cef9ade26f8bf66e808b7722efb0fd_1
183a0e5e6370   184ef4d127b4             "kube-controller-man…"   6 seconds ago       Up 5 seconds                 k8s_kube-controller-manager_kube-controller-manager-ecs-89475272-001_kube-system_fd4f9afb87e60d255e8e7aeec62a2914_1
23deee3bdc12   8e60ea3644d6             "kube-scheduler --au…"   6 seconds ago       Up 5 seconds                 k8s_kube-scheduler_kube-scheduler-ecs-89475272-001_kube-system_0a517741535b9a188d4e44b6382ca1a4_1
6294b5df643b   kubesphere/pause:3.4.1   "/pause"                 7 seconds ago       Up 6 seconds                 k8s_POD_kube-controller-manager-ecs-89475272-001_kube-system_fd4f9afb87e60d255e8e7aeec62a2914_1
c214ba0016a5   kubesphere/pause:3.4.1   "/pause"                 7 seconds ago       Up 6 seconds                 k8s_POD_kube-scheduler-ecs-89475272-001_kube-system_0a517741535b9a188d4e44b6382ca1a4_1
5df3235de17a   kubesphere/pause:3.4.1   "/pause"                 7 seconds ago       Up 6 seconds                 k8s_POD_kube-apiserver-ecs-89475272-001_kube-system_98cef9ade26f8bf66e808b7722efb0fd_1

重启kubelet

$ systemctl restart kubelet
$ systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since 二 2023-11-21 17:40:03 CST; 28s ago
     Docs: http://kubernetes.io/docs/
 Main PID: 61880 (kubelet)
    Tasks: 15
   Memory: 37.0M
   CGroup: /system.slice/kubelet.service
           └─61880 /usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=kubesphere/pause:3.4.1 --node-ip=* --hostname-override=ecs-89475272-001

更新 admin 配置

将新生成的 admin.conf 文件拷贝，替换 ~/.kube 目录下的 config 文件。

$ cp /etc/kubernetes/admin.conf ~/.kube/config
cp：是否覆盖"/root/.kube/config"？ y

重启calico

$ kubectl get ds -nkube-system
NAMESPACE     NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   calico-node    2         2         2       2            2           kubernetes.io/os=linux   23h
$ kubectl rollout restart ds calico-node -nkube-system
daemonset.apps/calico-node restarted

四、完成

完成以上操作后整个集群就可以正常通信了。