kubernetes 版本从 1.26.6 升级到 1.27.9

kubernetes版本的升级主要是升级kubeadm、kubectl和kubelet，升级应遵循以下原则：

1、apiserver保证版本最大，其它功能最多比apiserver小一个小版本号。

2、不能跨小版本升级。

3、更新前要修改 /etc/yum.repos.d/kubernetes.repo 文件中 baseurl 和 gpgkey 的URL到相应的升级版本。

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

使用下列命令列出可用版本

yum list --showduplicates kubeadm --disableexcludes=kubernetes

我的环境包括一个控制节点和二个业务节点，操作系统CentOS7。

一、升级第一个控制节点

1、升级kubeadm

yum install -y kubeadm-1.27.9-150500.1.1 --disableexcludes=kubernetes

kubeadm upgrade plan

kubeadm upgrade apply v1.27.9

kubectl drain k8s-m --ignore-daemonsets --force

yum install -y kubelet-1.27.9-150500.1.1 kubectl-1.27.9-150500.1.1 --disableexcludes=kubernetes

systemctl daemon-reload && systemctl restart kubelet

kubectl uncordon k8s-m

视情况升级 flannel

 kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

如还有其它控制节点与主节点稍有不同

kubeadm upgrade node

#execute on master control plane node
kubectl drain k8s-m-x --ignore-daemonsets

yum install -y kubelet-1.27.9-150500.1.1 kubectl-1.27.9-150500.1.1 --disableexcludes=kubernetes

systemctl daemon-reload && systemctl restart kubelet

#execute on master control plane node
kubectl uncordon k8s-m-x

2、升级业务节点（同样适用于剩余的其它控制节点）

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

yum install -y kubeadm-1.27.9-150500.1.1 --disableexcludes=kubernetes

kubeadm upgrade node

#在master节点执行
kubectl drain k8s-n-1 --ignore-daemonsets --delete-emptydir-data --force

yum install -y kubelet-1.27.9-150500.1.1 kubectl-1.27.9-150500.1.1 --disableexcludes=kubernetes

systemctl daemon-reload && systemctl restart kubelet

#在master节点执行
kubectl uncordon k8s-n-1

更新问题分享，其中一些网上根本找不到资料，难倒只有我这么幸运：（

1、从1.22.5升级到1.23.17的过程中不知我系统的问题还其它，当执行到 kubeadm upgrade plan 命令时会出现如下错误(已确认是我的问题，另一篇文章有更好的解决办法)

[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade/config] In order to upgrade, a ConfigMap called "kubeadm-config" in the kube-system namespace must exist.
[upgrade/config] Without this information, 'kubeadm upgrade' won't know how to configure your upgraded cluster.

[upgrade/config] Next steps:
        - OPTION 1: Run 'kubeadm config upload from-flags' and specify the same CLI arguments you passed to 'kubeadm init' when you created your control-plane.
        - OPTION 2: Run 'kubeadm config upload from-file' and specify the same config file you passed to 'kubeadm init' when you created your control-plane.
        - OPTION 3: Pass a config file to 'kubeadm upgrade' using the --config flag.

[upgrade/config] FATAL: the ConfigMap "kubeadm-config" in the kube-system namespace used for getting configuration information was not found
To see the stack trace of this error execute with --v=5 or higher

按照提示 kubectl -n kube-system get cm kubeadm-config -o yaml 是有配置的，不知为何提示找不到，我的解决办法是将该命令的输出另存为一个/etc/kubernetes/kubeadm-config.yaml文件，这里需要注意的是只把 ClusterConfiguration 段另存即可，否则会提示如下错误
invalid configuration for GroupVersionKind /v1, Kind=ConfigMap: kind and apiVersion is mandatory information that must be specified

配置文件准备好后执行如下命令来更新系统

kubeadm upgrade plan --config=/etc/kubernetes/kubeadm-config.yaml
kubeadm upgrade apply v1.23.17 --config=/etc/kubernetes/kubeadm-config.yaml

2、可能是我比较懒很久没更新了导致1.22已不再支持，在更新过程中出现如下错误

I0515 11:50:46.554140   19030 checks.go:859] pulling: k8s.gcr.io/coredns:v1.8.6
[preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:v1.8.6: output: time="2023-05-15T11:50:48+08:00" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"k8s.gcr.io/coredns:v1.8.6\": failed to resolve reference \"k8s.gcr.io/coredns:v1.8.6\": k8s.gcr.io/coredns:v1.8.6: not found"
, error: exit status 1

该问题是因为coredns镜像已经更改为另一个路径了，可以通过Tag方式来解决，我使用的containerd，命令如下

crictl pull k8s.gcr.io/coredns/coredns:v1.8.6
ctr -n k8s.io image tag k8s.gcr.io/coredns/coredns:v1.8.6 k8s.gcr.io/coredns:v1.8.6kube

集群更新后，执行如下指令切换成正确的镜像，否则Node也会找不到。

kubectl set image --namespace kube-system deployment.apps/coredns  coredns=k8s.gcr.io/coredns/coredns:v1.8.6

3、升级启动kubelet时的以下错误通过屏蔽 /var/lib/kubelet/kubeadm-flags.env 文件中相应参数解决

Jun 14 10:50:41 k8s-m kubelet[12310]: Error: failed to parse kubelet flag: unknown flag: --network-plugin

3、尽量不要更新最新次要版本，有可能会出现镜像下载不了的情况。

可参考官方升级文档：

Upgrading kubeadm clusters | Kubernetes

本文随新版本发布不定期更新。

吐槽一个这个编辑器，我只是按个ctrl+z，把我所有输入都取消了，而且不能恢复，真是可以的。