对由kubeadm安装的kubernetes集群进行升级
kubernetes版本升级应该是一件常见的事情,kubernetes在2015年7月21号发布1.0版本,到今年2019年已经到了1.15版本,而且版本的发布速度越来越快,对于DevOps人员来讲,kubernetes版本升级也就成了一个必备技能。本文会讲述将版本从 v1.10.x
升级到 v1.11.x
。升级主要包含二部分:kubeadm的升级,master节点和node节点上kubernetes 包(kubelet)的升级。
注意事项:
-
不能跨版本升级,只能从一个minor版本升级到下一个minor版本,比如从
v1.10.x
到v1.11.x
,不能从v1.9.x
直接到v1.11.x
。 -
在
v1.11
以后默认的DNS是CoreDNS
而不是kube-dns
,如果升级以后想继续用kube-dns
,则应该是升级的时候添加参数kubeadm upgrade apply v1.11.x --feature-gates=CoreDNS=false
kubeadm 升级
在master节点上,按照如下命令可以完成kubeadm的升级
$ export VERSION=$(curl -sSL https://dl.k8s.io/release/stable.txt)
$ export ARCH=amd64
$ curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
$ chmod a+rx /usr/bin/kubeadm
这个地方要特别注意,curl -sSL https://dl.k8s.io/release/stable.txt
会获取到kubernetes当前最稳定版本信息,目前为止是 v1.15.0
, 也就是意味着 kubeadm
的版本是 v1.15.0
。如果按照上述方案,升级之后的 kubeadm
是 v1.15.0
,然后在master节点上面执行 kubeadm upgrade plan
会出现如下错误
$ kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[upgrade/config] FATAL: unexpected error when reading kubeadm-config ConfigMap: ClusterConfiguration key value pair missing
原因是因为 kubeadm
的版本过高,可以通过降低版本来实现。(https://serverfault.com/questions/943696/fatal-unexpected-error-when-reading-kubeadm-config-configmap-clusterconfigurat
)
。我现在的版本是 v1.10.x
我要升级到 v1.11.x
, 在安装 kubeadm
的时候直接指定版本 v1.11.0
就可以解决这个问题
$ curl -sSL https://dl.k8s.io/release/v1.11.0/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
$ chmod a+rx /usr/bin/kubeadm
接着执行 kubeadm upgrade plan
命令
$ kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
I0710 03:35:06.733248 16862 feature_gate.go:230] feature gates: &{map[]}
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.10.11
[upgrade/versions] kubeadm version: v1.11.0
[upgrade/versions] Latest stable version: v1.15.0
[upgrade/versions] Latest version in the v1.10 series: v1.10.13
[upgrade/versions] WARNING: No recommended etcd for requested kubernetes version (v1.15.0)
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 4 x v1.10.4 v1.10.13
Upgrade to the latest version in the v1.10 series:
COMPONENT CURRENT AVAILABLE
API Server v1.10.11 v1.10.13
Controller Manager v1.10.11 v1.10.13
Scheduler v1.10.11 v1.10.13
Kube Proxy v1.10.11 v1.10.13
CoreDNS 1.1.3
Kube DNS 1.14.8
Etcd 3.1.12 3.1.12
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.10.13
____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 4 x v1.10.4 v1.15.0
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.10.11 v1.15.0
Controller Manager v1.10.11 v1.15.0
Scheduler v1.10.11 v1.15.0
Kube Proxy v1.10.11 v1.15.0
CoreDNS 1.1.3
Kube DNS 1.14.8
Etcd 3.1.12 N/A
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.15.0
Note: Before you can perform this upgrade, you have to update kubeadm to v1.15.0.
_____________________________________________________________________
接下来执行升级命令 kubeadm upgrade apply v1.11.0
。如果指定的版本高于 v1.11.x
, 比如 v1.12.x
就会如下错
[upgrade/version] FATAL: The --version argument is invalid due to these fatal errors:
- Specified version to upgrade to "v1.12.0" is too high; kubeadm can upgrade only 1 minor version at a time
- Specified version to upgrade to "v1.12.0" is at least one minor release higher than the kubeadm minor release (12 > 11). Such an upgrade is not supported
Please fix the misalignments highlighted above and try upgrading again
所以不可以跨minor版本升级,只能逐级升级。
升级master节点和node节点的kubernetes包(kubelet & kubeadm)
我们先来升级master节点,先用命令(kubectl drain $HOST --ignore-daemonsets
)将master节点设置为维护状态
$ kubectl drain k8stest01 --ignore-daemonsets --delete-local-data --force
node "k8stest01" cordoned
WARNING: Ignoring DaemonSet-managed pods: kube-flannel-ds-sxm2t, kube-proxy-bmg55, kube-prometheus-exporter-node-nj7nf
pod "coredns-78fcdf6894-ctvbl" evicted
node "k8stest01" drained
此时,如果可用 kubectl get nodes
查看一下master节点的状态
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8stest01 Ready,SchedulingDisabled master 203d v1.10.4
k8stest02 Ready <none> 203d v1.10.0
k8stest03 Ready <none> 203d v1.10.0
k8stest04 Ready <none> 203d v1.10.0
然后根据服务器的类型,选择相应的升级命令,我用的服务器是 ubuntu
,所以执行下面命令
$apt-get update
$apt-get upgrade -y kubelet kubeadm
上述命令安装的 kubelet & kubeadm
都是最新版本,为了集群相匹配,可以用如下命令安装指定的版本
$ export ARCH=amd64
$ curl -sSL https://dl.k8s.io/release/v1.11.0/bin/linux/${ARCH}/kubelet > /usr/bin/kubelet
$ chmod a+rx /usr/bin/kubelet
$ curl -sSL https://dl.k8s.io/release/v1.11.0/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
$ chmod a+rx /usr/bin/kubeadm
然后查看kubelet和kubeadm的版本
$ kubelet --version
Kubernetes v1.11.0
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
接着重启 kubelet
服务
$ systemctl restart kubelet
$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2019-07-10 09:18:33 EDT; 2h 8min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 29813 (kubelet)
Tasks: 21
Memory: 54.0M
CPU: 4min 8.792s
CGroup: /system.slice/kubelet.service
└─29813 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc
接着使用命令 kubectl uncordon k8stest01
将master节点k8stest01从维护状态调整为可调度状态,然后查看版本信息我们看到master节点的版本已经从 v1.10.x
升至 v1.11.x
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8stest01 Ready master 204d v1.11.0
k8stest02 Ready <none> 204d v1.10.0
k8stest03 Ready <none> 204d v1.10.0
k8stest04 Ready <none> 204d v1.10.0
至此,master节点上kubernetes的包(kubelet,kubeadm)已经升级完成。接下来升级node节点上的kubelet和kubeadm。node节点的升级和master的步骤一样,第一步也是要将node节点设置为维护状态 kubectl drain k8stest02 --ignore-daemonsets --delete-local-data --force
。接着升级kubelet和kubeadm,node节点比master多一个步骤,先执行如下命令
$ apt-get update
$ export ARCH=amd64
$ curl -sSL https://dl.k8s.io/release/v1.11.0/bin/linux/${ARCH}/kubelet > /usr/bin/kubelet
$ chmod a+rx /usr/bin/kubelet
$ curl -sSL https://dl.k8s.io/release/v1.11.0/bin/linux/${ARCH}/kubeadm > /usr/bin/kubeadm
$ chmod a+rx /usr/bin/kubeadm
然后执行 kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2)
$ kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2)
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
接着重启 kubelet
服务
$ systemctl restart kubelet
$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2019-07-10 09:18:33 EDT; 2h 8min ago
Docs: https://kubernetes.io/docs/home/
Main PID: 29813 (kubelet)
Tasks: 21
Memory: 54.0M
CPU: 4min 8.792s
CGroup: /system.slice/kubelet.service
└─29813 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc
接着使用命令 kubectl uncordon k8stest02
将node节点k8stest02从维护状态调整为可调度状态,然后查看版本信息
$ kubectl uncordon k8stest02
node/k8stest02 uncordoned
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8stest01 Ready master 204d v1.11.0
k8stest02 Ready <none> 204d v1.11.0
k8stest03 Ready <none> 204d v1.10.0
k8stest04 Ready <none> 204d v1.10.0
我们看到node节点k8stest02的版本已经从 v1.10.x
升至 v1.11.x
。其他节点都可以按照上述方法进行升级。升级完成的结果如下
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8stest01 Ready master 204d v1.11.0
k8stest02 Ready <none> 204d v1.11.0
k8stest03 Ready <none> 204d v1.11.0
k8stest04 Ready <none> 204d v1.11.0