目录
当我们要用到新版本的一些功能和特性的时候或者当前版本太旧无法满足需要的时候我们势必要对Kubernetes集群进行升级。
升级Master节点
腾空节点
kubectl drain master --ignore-daemonsets
[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 15m v1.22.0
node1 Ready <none> 14m v1.22.0
node2 Ready <none> 14m v1.22.0
[root@master ~]# kubectl drain master --ignore-daemonsets
node/master cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-xfmz2, kube-system/kube-proxy-gv47x
evicting pod kube-system/coredns-7f6cbbb7b8-v5hd7
evicting pod kube-system/coredns-7f6cbbb7b8-rp2th
pod/coredns-7f6cbbb7b8-v5hd7 evicted
pod/coredns-7f6cbbb7b8-rp2th evicted
node/master evicted
升级kubeadm
yum install -y kubelet-1.23.17 kubeadm-1.23.17 kubectl-1.23.17
sudo systemctl daemon-reload
sudo systemctl restart kubelet
验证升级计划
kubeadm upgrade plan
[root@master ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.22.0
[upgrade/versions] kubeadm version: v1.23.0
I1206 04:27:44.913702 15313 version.go:255] remote version is much newer: v1.28.4; falling back to: stable-1.23
[upgrade/versions] Target version: v1.23.17
[upgrade/versions] Latest version in the v1.22 series: v1.22.17
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 2 x v1.22.0 v1.22.17
1 x v1.23.0 v1.22.17
Upgrade to the latest version in the v1.22 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.22.0 v1.22.17
kube-controller-manager v1.22.0 v1.22.17
kube-scheduler v1.22.0 v1.22.17
kube-proxy v1.22.0 v1.22.17
CoreDNS v1.8.4 v1.8.6
etcd 3.5.0-0 3.5.1-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.22.17
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 2 x v1.22.0 v1.23.17
1 x v1.23.0 v1.23.17
Upgrade to the latest stable version:
COMPONENT CURRENT TARGET
kube-apiserver v1.22.0 v1.23.17
kube-controller-manager v1.22.0 v1.23.17
kube-scheduler v1.22.0 v1.23.17
kube-proxy v1.22.0 v1.23.17
CoreDNS v1.8.4 v1.8.6
etcd 3.5.0-0 3.5.1-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.23.17
Note: Before you can perform this upgrade, you have to update kubeadm to v1.23.17.
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
升级节点
kubeadm upgrade apply v1.23.17
[root@master ~]# kubeadm upgrade apply v1.23.17
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.23.17"
[upgrade/versions] Cluster version: v1.22.0
[upgrade/versions] kubeadm version: v1.23.17
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.23.17"...
Static pod: kube-apiserver-master hash: d7effe511a5e831f4033415992d35b15
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-master hash: fadc4a2c422c2731fad608bc36ebe3ee
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-06-04-31-02/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-master hash: fadc4a2c422c2731fad608bc36ebe3ee
Static pod: etcd-master hash: fadc4a2c422c2731fad608bc36ebe3ee
Static pod: etcd-master hash: 137cec5c581183ea4b778bae7143905d
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests1859383152"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-06-04-31-02/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-master hash: d7effe511a5e831f4033415992d35b15
Static pod: kube-apiserver-master hash: d7effe511a5e831f4033415992d35b15
Static pod: kube-apiserver-master hash: d7effe511a5e831f4033415992d35b15
Static pod: kube-apiserver-master hash: d7effe511a5e831f4033415992d35b15
Static pod: kube-apiserver-master hash: 3b409c5fb0f5e15e5b22a60e6de6385a
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-06-04-31-02/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 9a3f1caa5f95621c667c2d2e0796949d
Static pod: kube-controller-manager-master hash: 233725f24ab0fdceaed3247b43e35819
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-12-06-04-31-02/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: c77487dec29df94b978538919e5897eb
Static pod: kube-scheduler-master hash: 5baca7afd4a5a2d44d846da93480249e
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Applying label node-role.kubernetes.io/control-plane='' to Nodes with label node-role.kubernetes.io/master='' (deprecated)
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.23.17". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
解除节点保护
kubectl uncordon master
[root@master ~]# kubectl uncordon master
node/master uncordoned
[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 27m v1.23.17
node1 Ready <none> 26m v1.22.0
node2 Ready <none> 26m v1.22.0
升级Node节点
腾空节点
# 要升级哪个节点就腾空哪个节点
kubectl drain node1 --ignore-daemonsets
[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 27m v1.23.17
node1 Ready <none> 26m v1.22.0
node2 Ready <none> 26m v1.22.0
[root@master ~]# kubectl drain node1 --ignore-daemonsets
node/node1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-mv8t2, kube-system/kube-proxy-72ksg
evicting pod kube-system/coredns-6d8c4cb4d-mdf82
pod/coredns-6d8c4cb4d-mdf82 evicted
node/node1 drained
升级kubeadm
yum install -y kubelet-1.23.17 kubeadm-1.23.17 kubectl-1.23.17
sudo systemctl daemon-reload
sudo systemctl restart kubelet
升级节点
# 在需要升级的节点执行
kubeadm upgrade node
[root@node1 ~]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
解除节点保护
# 解除刚才升级的节点保护
kubectl uncordon node1
[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 33m v1.23.17
node1 Ready <none> 32m v1.23.17
node2 Ready <none> 32m v1.22.0
所有的node节点重复次步骤即可