k8s 1.23单Master版本containerd运行时版本升级【开发/测试环境】

升级前状态查看

[root@node1 modules-load.d]# kubectl get node
NAME    STATUS   ROLES                  AGE     VERSION
node1   Ready    control-plane,master   5h44m   v1.23.17
node2   Ready    <none>                 5h43m   v1.23.17
node3   Ready    <none>                 5h42m   v1.23.17

前置条件

#！！！非常重要
kubectl edit nodes node1
#需要修改kubeadm.alpha.kubernetes.io/cri-socket该值为containerd的sock文件
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock

master节点升级

[root@node1 modules-load.d]# yum  --showduplicates list kubeadm | grep 1.24
kubeadm.x86_64                       1.24.0-0                        kubernetes 
kubeadm.x86_64                       1.24.1-0                        kubernetes 
kubeadm.x86_64                       1.24.2-0                        kubernetes 
kubeadm.x86_64                       1.24.3-0                        kubernetes 
kubeadm.x86_64                       1.24.4-0                        kubernetes 
kubeadm.x86_64                       1.24.5-0                        kubernetes 
kubeadm.x86_64                       1.24.6-0                        kubernetes 
kubeadm.x86_64                       1.24.7-0                        kubernetes 
kubeadm.x86_64                       1.24.8-0                        kubernetes 
kubeadm.x86_64                       1.24.9-0                        kubernetes 
kubeadm.x86_64                       1.24.10-0                       kubernetes 
kubeadm.x86_64                       1.24.11-0                       kubernetes 
kubeadm.x86_64                       1.24.12-0                       kubernetes 
kubeadm.x86_64                       1.24.13-0                       kubernetes 
kubeadm.x86_64                       1.24.14-0                       kubernetes 
kubeadm.x86_64                       1.24.15-0                       kubernetes 
kubeadm.x86_64                       1.24.16-0                       kubernetes 
kubeadm.x86_64                       1.24.17-0                       kubernetes 

#升级到指定版本，大版本只能逐个按顺序升级，无法跳大版本
yum install kubeadm-1.24.17-0 -y

#查看升级计划
kubeadm upgrade plan
#输出信息如下
[root@node1 modules-load.d]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0228 22:11:12.113344    8010 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/dockershim.sock". Please update your configuration!
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.23.17
[upgrade/versions] kubeadm version: v1.24.17
I0228 22:11:19.835894    8010 version.go:256] remote version is much newer: v1.29.2; falling back to: stable-1.24
[upgrade/versions] Target version: v1.24.17
[upgrade/versions] Latest version in the v1.23 series: v1.23.17

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT        TARGET
kubelet     3 x v1.23.17   v1.24.17

Upgrade to the latest stable version:

COMPONENT                 CURRENT    TARGET
kube-apiserver            v1.23.17   v1.24.17
kube-controller-manager   v1.23.17   v1.24.17
kube-scheduler            v1.23.17   v1.24.17
kube-proxy                v1.23.17   v1.24.17
CoreDNS                   v1.8.6     v1.8.6
etcd                      3.5.6-0    3.5.6-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.24.17

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________


#执行升级计划
#！！！注意：升级过程集群可能无法正常进行变更操作，但是运行中的pod不受影响
kubeadm upgrade apply v1.24.17
#输出信息如下：
[root@node1 modules-load.d]# kubeadm upgrade apply v1.24.17
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0228 22:30:41.880098   21530 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/containerd/containerd.sock". Please update your configuration!
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.24.17"
[upgrade/versions] Cluster version: v1.23.17
[upgrade/versions] kubeadm version: v1.24.17
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.24.17" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests299697865"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-02-28-22-31-01/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-02-28-22-31-01/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-02-28-22-31-01/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Removing the deprecated label node-role.kubernetes.io/master='' from all control plane Nodes. After this step only the label node-role.kubernetes.io/control-plane='' will be present on control plane Nodes.
[upgrade/postupgrade] Adding the new taint &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} to all control plane Nodes. After this step both taints &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} and &Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,} should be present on control plane Nodes.
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.24.17". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

#更新kubelet\kubectl
yum  --showduplicates list kubelet | grep 1.24
yum  --showduplicates list kubectl | grep 1.24
yum install kubelet-1.24.17-0 kubectl-1.24.17-0 -y

#重启kubelet服务
systemctl daemon-reload
systemctl restart kubelet

#注意！！！！，此处升级完之后会报错如下
#node1 kubelet: Error: failed to parse kubelet flag: unknown flag: --network-plugin
#解决方法如下，去掉文件中的--network-plugin=cni参数
[root@node1 kubelet]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5"


#master节点升级完成，查看状态
[root@node1 kubelet]# kubectl get node
NAME    STATUS   ROLES           AGE   VERSION
node1   Ready    control-plane   15h   v1.24.17
node2   Ready    <none>          15h   v1.23.17
node3   Ready    <none>          15h   v1.23.17

worker节点升级

#升级组件
yum install kubelet-1.24.17-0 kubectl-1.24.17-0 -y

#注意！！！！，此处升级完之后会报错如下
#node1 kubelet: Error: failed to parse kubelet flag: unknown flag: --network-plugin
#解决方法如下，去掉文件中的--network-plugin=cni参数
[root@node1 kubelet]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5"


#重启服务
systemctl daemon-reload
systemctl restart kubelet

#升级完成，查看状态
[root@node1 kubelet]# kubectl get node
NAME    STATUS   ROLES           AGE   VERSION
node1   Ready    control-plane   15h   v1.24.17
node2   Ready    <none>          15h   v1.24.17
node3   Ready    <none>          15h   v1.24.17

注意点

需要修改/var/lib/kubelet/kubeadm-flags.env文件。