文章引导
前言
基础环境
升级准备
升级master节点
升级node节点
验证
前言
本章由Kubeadm来升级Kubernetes 集群从 1.19.9 版本 升级到 1.20.9 版本。但是不能跨版本升级，如：从1.19.9升级到1.21.9版本，只能一个版本的一个版本按顺序升级。如果从1.19.x升级到1.23.x需要升级4次，才能完成。升级流程为：1.升级master节点 2.升级工作节点
基础环境
| CPU架构 | 系统版本 | Docker版本 | K8S版本 |
|---|---|---|---|
| x86_64 | CentOS 7.8 | 19.03.12 | v1.19.9 |
| x86_64 | CentOS 7.8 | 19.03.12 | v1.19.9 |
| x86_64 | CentOS 7.8 | 19.03.12 | v1.19.9 |
升级准备
对于内网环境请在有外网的电脑上下载rpm包
- 使用国内yum源，这样可以避免下载不下来的问题，提高效率。
[root@ks8-master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=0
> repo_gpgcheck=0
> gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
> http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
> EOF
- 验证是否写入成功
[root@ks8-master ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
- 使用yum列出我们需要的Kubeadm的版本列表
[root@master yum.repos.d]# yum list --showduplicates kubeadm --disableexcludes=kubernetes | grep 1.20.9
kubeadm.x86_64 1.20.9-0 kubernetes
- 外网电脑直接安装就行
[root@master kubeadm]# yum install -y kubeadm-1.20.9-0
- 下载并查看kubeadm的rpm包，如果遇到下载下载一堆rpm包那么都需要一起打包到内网的master节点上，因为是依赖包，不然会安装不上（rpm包的名字看不懂不需要管）
[root@master home]# yum install --downloadonly --downloaddir=/home/kubeadm kubeadm-1.20.9-0 --disableexcludes=kubernetes
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.20.9-0 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=========================================================================================================================================
Package Arch Version Repository Size
=========================================================================================================================================
Installing:
kubeadm x86_64 1.20.9-0 kubernetes 8.3 M
Transaction Summary
=========================================================================================================================================
Install 1 Package
Total download size: 8.3 M
Installed size: 37 M
Background downloading packages, then exiting:
8c6b5ba8f467558ee1418d44e30310b7a8d463fc2d2da510e8aeeaf0edbed044-kubeadm-1.20.9-0.x86_64.rpm | 8.3 MB 00:00:00
exiting because "Download Only" specified
[root@master home]# ls -lh /home/kubeadm/
total 8.3M
-rw-r--r-- 1 root root 8.3M Jul 17 2021 8c6b5ba8f467558ee1418d44e30310b7a8d463fc2d2da510e8aeeaf0edbed044-kubeadm-1.20.9-0.x86_64.rpm
- 内网master安装kubeadm-1.20.9版本
[root@ks8-master ~]# yum -y install 8c6b5ba8f467558ee1418d44e30310b7a8d463fc2d2da510e8aeeaf0edbed044-kubeadm-1.20.9-0.x86_64.rpm
已加载插件：fastestmirror
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
正在检查 8c6b5ba8f467558ee1418d44e30310b7a8d463fc2d2da510e8aeeaf0edbed044-kubeadm-1.20.9-0.x86_64.rpm: kubeadm-1.20.9-0.x86_64
8c6b5ba8f467558ee1418d44e30310b7a8d463fc2d2da510e8aeeaf0edbed044-kubeadm-1.20.9-0.x86_64.rpm 将被安装
正在解决依赖关系
--> 正在检查事务
---> 软件包 kubeadm.x86_64.0.1.20.9-0 将被 安装
--> 解决依赖关系完成
依赖关系解决
=========================================================================================================================================
Package 架构 版本 源 大小
=========================================================================================================================================
正在安装:
kubeadm x86_64 1.20.9-0 /8c6b5ba8f467558ee1418d44e30310b7a8d463fc2d2da510e8aeeaf0edbed044-kubeadm-1.20.9-0.x86_64 37 M
事务概要
=========================================================================================================================================
安装 1 软件包
总计：37 M
安装大小：37 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : kubeadm-1.20.9-0.x86_64 1/1
验证中 : kubeadm-1.20.9-0.x86_64 1/1
已安装:
kubeadm.x86_64 0:1.20.9-0
完毕！
- 验证内外master节点的kubeadm是否升级成功
[root@ks8-master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9", GitCommit:"7a576bc3935a6b555e33346fd73ad77c925e9e4a", GitTreeState:"clean", BuildDate:"2021-07-15T21:00:30Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
- 因升级k8s集群的时候需要拉去镜像包，内网环境拉去不下来镜像和外网环境拉去不下来k8s镜像的解决办法，需要修改kubeadm-config的yaml文件使其拉去国内的升级镜像包）
把imageRepository的地址缓存阿里的地址，把imageRepository的值改为registry.aliyuncs.com/google_containers。然后:wq保存退出,不需要重启kubeadm，自动会更新
[root@ks8-master ~]# kubectl edit cm kubeadm-config -n kube-system #保存退出会输出一下提示，不需要重启kubeadm
configmap/kubeadm-config edited
##修改后的样子
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers #修改这一段
kind: ClusterConfiguration
kubernetesVersion: v1.19.9
- 验证升级计划，最后像这样提示就提示验证成功
[root@ks8-master ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.19.9
[upgrade/versions] kubeadm version: v1.20.9
W0616 10:53:46.447780 19848 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get "https://dl.k8s.io/release/stable.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0616 10:53:46.447852 19848 version.go:103] falling back to the local client version: v1.20.9
[upgrade/versions] Latest stable version: v1.20.9
[upgrade/versions] Latest stable version: v1.20.9
W0616 10:53:56.459770 19848 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.19.txt": Get "https://dl.k8s.io/release/stable-1.19.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0616 10:53:56.459815 19848 version.go:103] falling back to the local client version: v1.20.9
[upgrade/versions] Latest version in the v1.19 series: v1.20.9
[upgrade/versions] Latest version in the v1.19 series: v1.20.9
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
kubelet 4 x v1.19.9 v1.20.9
Upgrade to the latest version in the v1.19 series:
COMPONENT CURRENT AVAILABLE
kube-apiserver v1.19.9 v1.20.9
kube-controller-manager v1.19.9 v1.20.9
kube-scheduler v1.19.9 v1.20.9
kube-proxy v1.19.9 v1.20.9
CoreDNS 1.7.0 1.7.0
etcd 3.4.13-0 3.4.13-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.20.9
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
升级master节点
- 因为升级会去拉
kube-proxy、kube-controller-manager、kube-apiserver、kube-scheduler这四个镜像，所以在内网环境下，需要提前准备镜像。在外网的电脑上直接使用docker pull以下镜像，然后传到内网master节点上。（外网的k8s可以忽略这一步）
因为kubeadm更换了镜像源，所以在内网升级的时候会直接使用下列的镜像，拉去镜像的时候会先验证本地是否有镜像，如果有就会直接引用
[root@ks8-master ~]# docker images |grep 20
registry.aliyuncs.com/google_containers/kube-proxy v1.20.9 8dbf9a6aa186 23 months ago 99.7MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.20.9 295014c114b3 23 months ago 47.3MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.20.9 0d0d57e4f64c 23 months ago 122MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.20.9 eb07fd4ad3b4 23 months ago 116MB
- 外网开始升级，外网环境应该一会就会提示这个。如果不升级etcd那么加上
--etcd-upgrade=false
root@ks8-master ~]# kubeadm upgrade apply v1.20.9
##不升级etcd示例： kubeadm upgrade apply v1.20.9 --etcd-upgrade=false
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.20.9". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
- 如下图，在内网环境遇到镜像拉去不下来，就说明需要对于的版本镜像没有，如果master节点上有这个镜像只是镜像版本不同，那么请使用
docker tag命令使其改成下列提示需要的镜像标签。
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_container/pause:3.2: output: Error response from daemon: Get https://registry.aliyuncs.com/v2/: dial tcp: lookup registry.aliyuncs.com on [::1]:53: read udp [::1]:56670->[::1]:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_container/etcd:3.4.13-0: output: Error response from daemon: Get https://registry.aliyuncs.com/v2/: dial tcp: lookup registry.aliyuncs.com on [::1]:53: read udp [::1]:42806->[::1]:53: read: connect ion refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.aliyuncs.com/google_container/coredns:1.7.0: output: Error response from daemon: Get https://registry.aliyuncs.com/v2/: dial tcp: lookup registry.aliyuncs.com on [::1]:53: read udp [::1]:36907->[::1]:53: read: connect ion refused
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
- 内网开始升级，等待升级成功。
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.20.9". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
- 验证版本，Server端代表k8s，显示
v1.20.9版本，代表升级成功。client端代表kubectl
[root@ks8-master ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9", GitCommit:"7a576bc3935a6b555e33346fd73ad77c925e9e4a", GitTreeState:"clean", BuildDate:"2021-07-15T20:56:38Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.27) and server (1.20) exceeds the supported minor version skew of +/-1
- 验证集群版本，看到master的版本已经更变。如没有改变请重启kubelet。
[root@ks8-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node1 Ready <none> 30d v1.19.9
k8s-node2 Ready <none> 30d v1.19.9
k8s-node3 Ready <none> 26d v1.19.9
ks8-master Ready control-plane,master 30d v1.20.9
对于其它控制面节点与第一个控制面节点相同，但是使用：kubeadm upgrade node
升级node节点
- 按照升级master节点一样，先安装升级kubeadm。
事务概要
==============================================================================================================================
安装 10 软件包
总计：276 M
安装大小：276 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : libnetfilter_cttimeout-1.0.0-7.el7.x86_64 1/10
正在安装 : kubectl-1.27.3-0.x86_64 2/10
正在安装 : cri-tools-1.26.0-0.x86_64 3/10
正在安装 : libnetfilter_queue-1.0.2-2.el7_2.x86_64 4/10
正在安装 : libnetfilter_cthelper-1.0.0-11.el7.x86_64 5/10
正在安装 : conntrack-tools-1.4.4-7.el7.x86_64 6/10
正在安装 : socat-1.7.3.2-2.el7.x86_64 7/10
正在安装 : kubernetes-cni-1.2.0-0.x86_64 8/10
正在安装 : kubelet-1.27.3-0.x86_64 9/10
正在安装 : kubeadm-1.20.9-0.x86_64 10/10
验证中 : socat-1.7.3.2-2.el7.x86_64 1/10
验证中 : libnetfilter_cthelper-1.0.0-11.el7.x86_64 2/10
验证中 : conntrack-tools-1.4.4-7.el7.x86_64 3/10
验证中 : libnetfilter_queue-1.0.2-2.el7_2.x86_64 4/10
验证中 : cri-tools-1.26.0-0.x86_64 5/10
验证中 : kubernetes-cni-1.2.0-0.x86_64 6/10
验证中 : kubeadm-1.20.9-0.x86_64 7/10
验证中 : kubectl-1.27.3-0.x86_64 8/10
验证中 : libnetfilter_cttimeout-1.0.0-7.el7.x86_64 9/10
验证中 : kubelet-1.27.3-0.x86_64 10/10
已安装:
conntrack-tools.x86_64 0:1.4.4-7.el7 cri-tools.x86_64 0:1.26.0-0
kubeadm.x86_64 0:1.20.9-0 kubectl.x86_64 0:1.27.3-0
kubelet.x86_64 0:1.27.3-0 kubernetes-cni.x86_64 0:1.2.0-0
libnetfilter_cthelper.x86_64 0:1.0.0-11.el7 libnetfilter_cttimeout.x86_64 0:1.0.0-7.el7
libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7
完毕！
- 验证kubeadm版本
[root@k8s-node3 kubadmin-update]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9", GitCommit:"7a576bc3935a6b555e33346fd73ad77c925e9e4a", GitTreeState:"clean", BuildDate:"2021-07-15T21:00:30Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
- 执行
kubeadm upgrade node会升级本地的 kubelet 配置，检测是不是为工作节点。检测成功就可以进行下一步
[root@k8s-node3 kubadmin-update]# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
- 腾空节点，在master节点上执行。将节点标记为不可调度并驱逐所有负载，准备节点的维护。
[root@ks8-master ~]# kubectl drain k8s-node3 --ignore-daemonsets
node/k8s-node3 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-s7zx4, kube-system/kube-proxy-8gfdr
evicting pod default/zk-2
pod/zk-2 evicted
node/k8s-node3 drained
- 升级 kubelet 和 kubectl，外网电脑直接
yum -y install kubelet-1.20.9-0 kubectl-1.20.9-0下载
外网电脑上下载升级的rpm包。
yum install --downloadonly --downloaddir=/root/kubadmin-update kubelet-1.20.9-0 kubectl-1.20.9-0
已加载插件：fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* epel: mirrors.bfsu.edu.cn
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
kubernetes | 1.4 kB 00:00:00
kubernetes/primary | 132 kB 00:00:00
kubernetes 980/980
......
==============================================================================================================================
Package 架构 版本 源 大小
==============================================================================================================================
正在安装:
kubectl x86_64 1.20.9-0 kubernetes 8.5 M
kubelet x86_64 1.20.9-0 kubernetes 20 M
为依赖而安装:
conntrack-tools x86_64 1.4.4-7.el7 base 187 k
kubernetes-cni x86_64 1.2.0-0 kubernetes 17 M
libnetfilter_cthelper x86_64 1.0.0-11.el7 base 18 k
libnetfilter_cttimeout x86_64 1.0.0-7.el7 base 18 k
libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k
socat x86_64 1.7.3.2-2.el7 base 290 k
事务概要
==============================================================================================================================
安装 2 软件包 (+6 依赖软件包)
总计：46 M
总下载量：8.5 M
安装大小：198 M
Background downloading packages, then exiting:
c968b9ca8bd22f047f56a929184d2b0ec8eae9c0173146f2706cec9e24b5fefb-kubectl-1.20.9-0.x86_64.rpm | 8.5 MB 00:00:37
exiting because "Download Only" specified
- 内网安装使用
rpm命令来安装，然后重载配置，重启kubelet。
[root@k8s-node3 home]# rpm -ivh 02431d76ab73878211a6052a2fded564a3a2ca96438974e4b0baffb0b3cb883a-kubelet-1.20.9-0.x86_64.rpm
警告：02431d76ab73878211a6052a2fded564a3a2ca96438974e4b0baffb0b3cb883a-kubelet-1.20.9-0.x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID 3e1ba8d5: NOKEY
准备中... ################################# [100%]
正在升级/安装...
1:kubelet-1.20.9-0 ################################# [100%]
[root@k8s-node3 home]# rpm -ivh c968b9ca8bd22f047f56a929184d2b0ec8eae9c0173146f2706cec9e24b5fefb-kubectl-1.20.9-0.x86_64.rpm
警告：c968b9ca8bd22f047f56a929184d2b0ec8eae9c0173146f2706cec9e24b5fefb-kubectl-1.20.9-0.x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID 3e1ba8d5: NOKEY
准备中... ################################# [100%]
正在升级/安装...
1:kubectl-1.20.9-0 ################################# [100%]
[root@k8s-node3 home]# rpm -qa | grep kube*
kubernetes-cni-1.2.0-0.x86_64
kubelet-1.20.9-0.x86_64
kubeadm-1.20.9-0.x86_64
kubectl-1.20.9-0.x86_64
[root@k8s-node3 home]# systemctl daemon-reload
[root@k8s-node3 home]# systemctl restart kubelet
- 取消去节点的排空，master节点上执行。
[root@ks8-master ~]# kubectl uncordon k8s-node3
node/k8s-node3 uncordoned
验证
通过master节点验证，工作节点完成升级。一个工作节点和一个master节点就完成了升级。[root@ks8-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node1 Ready <none> 30d v1.19.9
k8s-node2 Ready <none> 30d v1.19.9
k8s-node3 Ready <none> 26d v1.20.9
ks8-master Ready control-plane,master 30d v1.20.9
2162
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
