Kubernetes集群更换证书（正常更新方法、和更新证书为99年）

cowboy-杰洛

已于 2022-12-14 16:18:31 修改

阅读量6k

点赞数 2

分类专栏： K8S学习日志文章标签： kubernetes 运维

于 2022-12-14 16:17:05 首次发布

本文链接：https://blog.csdn.net/m0_59308369/article/details/128311480

版权

K8S学习日志专栏收录该内容

1 篇文章 0 订阅

订阅专栏

1. 前言

1.2检查证书是否过期

2. ！！！备份原有集群配置文件

3. 官方更新方式，更新证书（有效期一年）

3.2测试集群是否正常

4. 使用开源组件将证书有效期设置为99年（建议初始化集群时）

1. 前言

配置信息： ubuntu18.04 k8s集群1.22.0。

更新证书后: 会导致contexts丢失（上下文） kubectl config get-contexts

更新证书

今天使用k8s集群发现报错。

root@k8s-master:~# kubectl get nodes
The connection to the server <master>:6443 was refused - did you specify the right host or port?

查看kubelet状态，发现无法启动，查看docker服务，发现api-server也没用启动

root@k8s-master:~# systemctl status kubelet.service 
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2022-12-14 09:25:38 CST; 3s ago
     Docs: https://kubernetes.io/docs/home/
  Process: 4364 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 4364 (code=exited, status=1/FAILURE)
root@k8s-master:~# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

查看日志，发现报错找不到 bootstrap-kubelet.conf

root@k8s-master:~# tail -f 20 /var/log/syslog 
tail: cannot open '20' for reading: No such file or directory
==> /var/log/syslog <==
Dec 14 09:28:22 k8s-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
Dec 14 09:28:22 k8s-master kubelet[5166]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 14 09:28:22 k8s-master kubelet[5166]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 14 09:28:22 k8s-master systemd[1]: Started Kubernetes systemd probe.
Dec 14 09:28:22 k8s-master kubelet[5166]: I1214 09:28:22.095203    5166 server.go:440] "Kubelet version" kubeletVersion="v1.22.0"
Dec 14 09:28:22 k8s-master kubelet[5166]: I1214 09:28:22.095517    5166 server.go:868] "Client rotation is on, will bootstrap in background"
Dec 14 09:28:22 k8s-master kubelet[5166]: E1214 09:28:22.096812    5166 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2022-12-08 06:32:35 +0000 UTC
Dec 14 09:28:22 k8s-master kubelet[5166]: E1214 09:28:22.096858    5166 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
Dec 14 09:28:22 k8s-master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Dec 14 09:28:22 k8s-master systemd[1]: kubelet.service: Failed with result 'exit-code'.
Dec 14 09:28:32 k8s-master systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Dec 14 09:28:32 k8s-master systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 59.
Dec 14 09:28:32 k8s-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Dec 14 09:28:32 k8s-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
Dec 14 09:28:32 k8s-master kubelet[5215]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 14 09:28:32 k8s-master kubelet[5215]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 14 09:28:32 k8s-master systemd[1]: Started Kubernetes systemd probe.
Dec 14 09:28:32 k8s-master kubelet[5215]: I1214 09:28:32.342032    5215 server.go:440] "Kubelet version" kubeletVersion="v1.22.0"
Dec 14 09:28:32 k8s-master kubelet[5215]: I1214 09:28:32.342792    5215 server.go:868] "Client rotation is on, will bootstrap in background"
Dec 14 09:28:32 k8s-master kubelet[5215]: E1214 09:28:32.345247    5215 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2022-12-08 06:32:35 +0000 UTC
Dec 14 09:28:32 k8s-master kubelet[5215]: E1214 09:28:32.345538    5215 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
Dec 14 09:28:32 k8s-master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Dec 14 09:28:32 k8s-master systemd[1]: kubelet.service: Failed with result 'exit-code'.
Dec 14 09:28:42 k8s-master systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Dec 14 09:28:42 k8s-master systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 60.
Dec 14 09:28:42 k8s-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Dec 14 09:28:42 k8s-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
Dec 14 09:28:42 k8s-master kubelet[5262]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 14 09:28:42 k8s-master kubelet[5262]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 14 09:28:42 k8s-master systemd[1]: Started Kubernetes systemd probe.
Dec 14 09:28:42 k8s-master kubelet[5262]: I1214 09:28:42.601131    5262 server.go:440] "Kubelet version" kubeletVersion="v1.22.0"
Dec 14 09:28:42 k8s-master kubelet[5262]: I1214 09:28:42.601981    5262 server.go:868] "Client rotation is on, will bootstrap in background"
Dec 14 09:28:42 k8s-master kubelet[5262]: E1214 09:28:42.604829    5262 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2022-12-08 06:32:35 +0000 UTC
Dec 14 09:28:42 k8s-master kubelet[5262]: E1214 09:28:42.604871    5262 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
Dec 14 09:28:42 k8s-master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Dec 14 09:28:42 k8s-master systemd[1]: kubelet.service: Failed with result 'exit-code'.

kubelet.conf (在 TLS 引导时名称为 bootstrap-kubelet.conf)，文件里有一个引导令牌或内嵌的客户端证书。每个节点都有

root@k8s-master:/etc# ll /etc/kubernetes/kubelet.conf 
-rw------- 1 root root 1979 Dec  8  2021 /etc/kubernetes/kubelet.conf

1.2检查证书是否过期

root@k8s-master:/etc# openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -text |grep Not
            Not Before: Dec  8 06:32:31 2021 GMT
            Not After : Dec  6 06:32:31 2031 GMT

root@k8s-master:/etc# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep Not
            Not Before: Dec  8 06:32:31 2021 GMT
            Not After : Dec  8 06:32:32 2022 GMT

root@k8s-master:/etc# kubeadm certs check-expiration 
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 08, 2022 06:32 UTC   <invalid>                               no      
apiserver                  Dec 08, 2022 06:32 UTC   <invalid>       ca                      no      
apiserver-etcd-client      Dec 08, 2022 06:32 UTC   <invalid>       etcd-ca                 no      
apiserver-kubelet-client   Dec 08, 2022 06:32 UTC   <invalid>       ca                      no      
controller-manager.conf    Dec 08, 2022 06:32 UTC   <invalid>                               no      
etcd-healthcheck-client    Dec 08, 2022 06:32 UTC   <invalid>       etcd-ca                 no      
etcd-peer                  Dec 08, 2022 06:32 UTC   <invalid>       etcd-ca                 no      
etcd-server                Dec 08, 2022 06:32 UTC   <invalid>       etcd-ca                 no      
front-proxy-client         Dec 08, 2022 06:32 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf             Dec 08, 2022 06:32 UTC   <invalid>                               no

发现CA证书尚未过期，apiserver证书过期、和集群证书全部过期。

命令参数解释：

第一行代码：查询CA证书过期信息

第二行代码：查询apiserver证书过期信息

-x509 表示输出证书

-in 需要进行处理的PEM格式的证书

-noout不打印密钥key数据

-test：text显示格式

## kubeadm init 初始化集群的时候会生成有效期为一年的集群证书。

2. ！！！备份原有集群配置文件

mkdir ~/confirm                               #备份源集群文件
cp -rf /etc/kubernetes/ ~/confirm/    
ll ~/confirm/kubernetes/

mkdir confirm/data_etcd                       #备份ETCD
cp -rf /var/lib/etcd/* ~/confirm/data_etcd  
            
cp /usr/bin/kubeadm /usr/bin/kubeadm.bak

3. 官方更新方式，更新证书（有效期一年）

kubeadm certs renew all                       #更新所有证书
kubeadm certs check-expiration                #检查集群证书信息

kubeadm alpha certs renew all更新的集群证书，但不会更新 kubelet.conf 的证书的
所以现在启动kubelet是失败的。

vim /etc/kubernetes/admin.conf   将user以及下面的经过base64编码的值  （17行开始）
 拷贝至 kubelet.conf

将nodes节点上的/etc/kubernetes/kubelet.conf 替换成master节点上的 kubelet.conf
之后重启kubelet服务。


systemctl restart docker.service && systemctl restart kubelet.service

admin.conf 这个文件包含集群的地址，admin用户及各种证书的密钥的信息，创建kubernetes集群时，这个文件里的admin用户已经被授予最大权限（通过这个文件找寻API server→资源的创建和访问）

kubele.conf

3.2测试集群是否正常

root@k8s-master:~# kubectl get nodes
error: You must be logged in to the server (Unauthorized)

报错原因：没将更新admin.conf文件配到当前用户变量里

root@k8s-master:~# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? y
root@k8s-master:~# kubectl get pods
NAME                                      READY   STATUS    RESTARTS       AGE
front-end-6f94965fd9-vsvz5                1/1     Running   0              30m
guestbook-86bb8f5bc9-8pqcl                1/1     Running   0              30m
guestbook-86bb8f5bc9-d9tph                1/1     Running   0              30m
nfs-client-provisioner-56dd5765dc-sdsxh   1/1     Running   0              30m
task-2-ds-kh94m                           1/1     Running   5 (354d ago)   355d
task-2-ds-v5dvj                           1/1     Running   4 (354d ago)   355d

4. 使用开源组件将证书有效期设置为99年（建议初始化集群时）

下载相关组件：

Golang 官网下载地址：golang官网
打开官网下载地址选择对应的系统版本, 复制下载链接

https://studygolang.com/dl/golang/go1.16.5.linux-amd64.tar.gz

https://codeload.github.com/kubernetes/kubernetes/zip/refs/tags/v1.22.0

下载与自己当前k8s集群版本相同的文件

apt-get install gcc automake autoconf libtool make #安装编译需要的组件


tar zxf go1.16.5.linux-amd64.tar.gz -C /usr/local/

vim /etc/profile
export GOROOT=/usr/local/go
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/go

source /etc/profile

root@k8s-master:~/test# go version 
go version go1.16.5 linux/amd64
root@k8s-master:~/test# go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.5"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4057545257=/tmp/go-build -gno-record-gcc-switches"

解压编辑源码包

修改证书时间：CertificateValidity = time.Hour * 24 * 365 * 100

修改：NotAfter: now.Add(duration365d * 100).UTC(),

unzip kubernetes-1.22.0\(1\).zip 
cd kubernetes-1.22.0/
vim cmd/kubeadm/app/constants/constants.go              #搜索CertificateValidity
const (
        // KubernetesDir is the directory Kubernetes owns for storing various configuration files
        KubernetesDir = "/etc/kubernetes"
        // ManifestsSubDirName defines directory name to store manifests
        ManifestsSubDirName = "manifests"
        // TempDirForKubeadm defines temporary directory for kubeadm
        // should be joined with KubernetesDir.
        TempDirForKubeadm = "tmp"

        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        CertificateValidity = time.Hour * 24 * 365 * 100

        // CACertAndKeyBaseName defines certificate authority base name
        CACertAndKeyBaseName = "ca"
        // CACertName defines certificate name
        CACertName = "ca.crt"

-------------------
vim staging/src/k8s.io/client-go/util/cert/cert.go      #搜索KeyUsageDigitalSignatur
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
        now := time.Now()
        tmpl := x509.Certificate{
                SerialNumber: new(big.Int).SetInt64(0),
                Subject: pkix.Name{
                        CommonName:   cfg.CommonName,
                        Organization: cfg.Organization,
                },
                DNSNames:              []string{cfg.CommonName},
                NotBefore:             now.UTC(),
                NotAfter:              now.Add(duration365d * 100).UTC(),
                KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
                BasicConstraintsValid: true,
                IsCA:                  true,
        }

编译安装

root@k8s-master:~/test/kubernetes-1.22.0# make WHAT=cmd/kubeadm
+++ [1214 15:37:11] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
Generating prerelease lifecycle code for 27 targets
+++ [1214 15:37:22] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
Generating deepcopy code for 234 targets
+++ [1214 15:37:29] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
Generating defaulter code for 93 targets
+++ [1214 15:37:37] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
Generating conversion code for 128 targets
+++ [1214 15:37:53] Building go targets for linux/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [1214 15:38:09] Building go targets for linux/amd64:
    cmd/kubeadm

root@k8s-master:~/test/kubernetes-1.22.0# ll _output/bin/
total 86596
drwxr-xr-x 2 root root     4096 Dec 14 15:39 ./
drwxr-xr-x 3 root root     4096 Dec 14 15:37 ../
-rwxr-xr-x 1 root root  7540736 Dec 14 15:37 conversion-gen*
-rwxr-xr-x 1 root root  7204864 Dec 14 15:37 deepcopy-gen*
-rwxr-xr-x 1 root root  7233536 Dec 14 15:37 defaulter-gen*
-rwxr-xr-x 1 root root  3555103 Dec 14 15:37 go2make*
-rwxr-xr-x 1 root root 45817856 Dec 14 15:39 kubeadm*
-rwxr-xr-x 1 root root 10141696 Dec 14 15:38 openapi-gen*
-rwxr-xr-x 1 root root  7172096 Dec 14 15:37 prerelease-lifecycle-gen*

将新生成的kubeadm进行替换，并更新证书

cp /root/test/kubernetes-1.22.0/_output/bin/kubeadm  /usr/bin/kubeadm
kubeadm certs renew all

[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Nov 20, 2122 07:47 UTC   99y                                     no      
apiserver                  Nov 20, 2122 07:47 UTC   99y             ca                      no      
apiserver-etcd-client      Nov 20, 2122 07:47 UTC   99y             etcd-ca                 no      
apiserver-kubelet-client   Nov 20, 2122 07:47 UTC   99y             ca                      no      
controller-manager.conf    Nov 20, 2122 07:47 UTC   99y                                     no      
etcd-healthcheck-client    Nov 20, 2122 07:47 UTC   99y             etcd-ca                 no      
etcd-peer                  Nov 20, 2122 07:47 UTC   99y             etcd-ca                 no      
etcd-server                Nov 20, 2122 07:47 UTC   99y             etcd-ca                 no      
front-proxy-client         Nov 20, 2122 07:47 UTC   99y             front-proxy-ca          no      
scheduler.conf             Nov 20, 2122 07:47 UTC   99y                                     no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 06, 2031 06:32 UTC   8y              no      
etcd-ca                 Dec 06, 2031 06:32 UTC   8y              no      
front-proxy-ca          Dec 06, 2031 06:32 UTC   8y              no

vim /etc/kubernetes/admin.conf   将user以及下面的经过base64编码的值  （17行开始）
 拷贝至 kubelet.conf

将nodes节点上的/etc/kubernetes/kubelet.conf 替换成master节点上的 kubelet.conf
之后重启kubelet服务。


systemctl restart docker.service && systemctl restart kubelet.service

4.2验证测试集群

root@k8s-master:~# kubectl get nodes
error: You must be logged in to the server (Unauthorized)

报错原因：没将更新admin.conf文件配到当前用户变量里

root@k8s-master:~# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? y
root@k8s-master:/etc/kubernetes# kubectl get nodes
NAME         STATUS   ROLES                  AGE    VERSION
k8s-master   Ready    control-plane,master   371d   v1.22.0
k8s-node1    Ready    <none>                 371d   v1.22.0
k8s-node2    Ready    <none>                 371d   v1.22.0

参考链接：

PKI 证书和要求 | Kubernetes

使用 kubeadm 进行证书管理 | Kubernetes

kubelet 证书轮换失败的解决方案_米开朗基杨的博客-CSDN博客

kubeadm修改证书时间为99年 - 简书 (jianshu.com)

cowboy-杰洛

关注

2
点赞
踩
13

收藏

觉得还不错? 一键收藏
1
评论
Kubernetes集群更换证书（正常更新方法、和更新证书为99年）

今天使用k8s集群发现报错。查看kubelet状态，发现无法启动，查看docker服务，发现api-server也没用启动enabled;3s ago查看日志，发现报错找不到 bootstrap-kubelet.conf。
复制链接

扫一扫