升级kubeadm安装部署的kubernetes集群

kubeadm只支持跨一个次要版本的升级，eg从1.19升级到1.20，而不能直接升级到1.21

1、确定要升级到哪个版本
    yum list --showduplicates kubeadm --disableexcludes=kubernetes
    # 在列表中查找最新的 1.26 版本
    # 它看起来应该是 1.26.x-0，其中 x 是最新的补丁版本

2、一般情况下，我们都会有多个控制节点，从代理层上摘除一个控制节点，然后进行升级操作，我们从1.19 升级到1.20
    yum install -y kubeadm-1.20.15-0 --disableexcludes=kubernetes
    验证下载操作正常，并且 kubeadm 版本正确
    kubeadm version
    验证升级计划
    kubeadm upgrade plan

3、验证升级计划：
    [root@master1 pki]# kubeadm upgrade plan
    [upgrade/config] Making sure the configuration is correct:
    [upgrade/config] Reading configuration from the cluster...
    [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
    [preflight] Running pre-flight checks.
    [upgrade] Running cluster health checks
    [upgrade] Fetching available versions to upgrade to
    [upgrade/versions] Cluster version: v1.19.16
    [upgrade/versions] kubeadm version: v1.20.15
    I0321 15:16:09.676407 1464931 version.go:254] remote version is much newer: v1.26.3; falling back to: stable-1.20
    [upgrade/versions] Latest stable version: v1.20.15
    [upgrade/versions] Latest stable version: v1.20.15
    [upgrade/versions] Latest version in the v1.19 series: v1.19.16
    [upgrade/versions] Latest version in the v1.19 series: v1.19.16
    
    Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
    COMPONENT   CURRENT        AVAILABLE
    kubelet     5 x v1.19.16   v1.20.15
    
    Upgrade to the latest stable version:
    
    COMPONENT                 CURRENT    AVAILABLE
    kube-apiserver            v1.19.16   v1.20.15
    kube-controller-manager   v1.19.16   v1.20.15
    kube-scheduler            v1.19.16   v1.20.15
    kube-proxy                v1.19.16   v1.20.15
    CoreDNS                   1.7.0      1.7.0
    
    You can now apply the upgrade by executing the following command:
    
        kubeadm upgrade apply v1.20.15
    
    _____________________________________________________________________
    
    
    The table below shows the current state of component configs as understood by this version of kubeadm.
    Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
    resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
    upgrade to is denoted in the "PREFERRED VERSION" column.
    
    API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
    kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
    kubelet.config.k8s.io     v1beta1           v1beta1             no
    _____________________________________________________________________

4、拉取所有依赖的镜像，推送到自建的镜像仓库
    1）查看所有依赖的镜像
    kubeadm config images list 
    2）从阿里云仓库拉取所有依赖的镜像
    拉取：
        docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.15
        docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.15
        docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.15
        docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.20.15
        docker pull registry.aliyuncs.com/google_containers/pause:3.2
        docker pull registry.aliyuncs.com/google_containers/etcd:3.4.13-0
        docker pull registry.aliyuncs.com/google_containers/coredns:1.7.0

    改tag：
        docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.20.15  xctest.com/kubernetes_1_18_9/kube-apiserver:v1.20.15
        docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.20.15  xctest.com/kubernetes_1_18_9/        kube-controller-manager:v1.20.15
        docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.20.15  xctest.com/kubernetes_1_18_9/        kube-scheduler:v1.20.15
        docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.20.15  xctest.com/kubernetes_1_18_9/kube-proxy:v1.20.15
        docker tag registry.aliyuncs.com/google_containers/pause:3.2  xctest.com/kubernetes_1_18_9/pause:3.2
        docker tag registry.aliyuncs.com/google_containers/etcd:3.4.13-0  xctest.com/kubernetes_1_18_9/etcd:3.4.13-0
        docker tag registry.aliyuncs.com/google_containers/coredns:1.7.0  xctest.com/kubernetes_1_18_9/coredns:1.7.0

    push：
        docker push xctest.com/kubernetes_1_18_9/kube-apiserver:v1.20.15
        docker push xctest.com/kubernetes_1_18_9/kube-controller-manager:v1.20.15
        docker push xctest.com/kubernetes_1_18_9/kube-scheduler:v1.20.15
        docker push xctest.com/kubernetes_1_18_9/kube-proxy:v1.20.15
        docker push xctest.com/kubernetes_1_18_9/pause:3.2
        docker push xctest.com/kubernetes_1_18_9/etcd:3.4.13-0
        docker push xctest.com/kubernetes_1_18_9/coredns:1.7.0
5、选择要升级到的目标版本，运行合适的命令。例如：
    kubeadm upgrade apply v1.20.15
    一旦该命令结束，你应该会看到：
        [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.20.15". Enjoy!
        [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
6、手动升级你的 CNI 驱动插件(建议最后在升级CNI的组件)
    你的容器网络接口（CNI）驱动应该提供了程序自身的升级说明。 参阅插件页面查找你的 CNI 驱动， 并查看是否需要其他升级步骤。
    如果插件为calico，首先我们要看下当前calico的版本，从calico官网查看calico的版本支持的kubernetes版本的范围（https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#kubernetes-requirements）
    升级calico操作：找到calico的安装方式的yaml文件，准备一份老的版本的配置和新的版本的calico的配置文件，对比一下差异性，判断是否可以直接apply，kubectl apply -f calico-new.yaml,观察calico相关的pod，整个过程不会对线上的服务的访问路由造成影响
    如果 CNI 驱动作为 DaemonSet 运行，则在其他控制平面节点上不需要此步骤
7、升级其它控制节点
    1）摘除其它控制节点的流量，更换到第一个节点,其余控制节点开始执行如下操作：
        yum install -y kubeadm-1.20.15-0 --disableexcludes=kubernetes
        kubeadm upgrade node
        更新其余控制节点时master会报错：
            Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
            I0321 09:37:30.725913       1 server.go:632] external host was not specified, using 10.136.158.8
            Error: [service-account-issuer is a required flag, --service-account-signing-key-file and --service-account-issuer are required flags]
        根据报错信息，kube-apiserver需要两个flag，查看第一个master节点/etc/kubernetes/manifests/kube-apiserver.yaml，将flag copy到其它控制节点中，问题即可解决
    2)升级所有master节点的kubelet 和 kubectl
        yum install -y kubelet-1.20.15-0 kubectl-1.20.15-0 --disableexcludes=kubernetes
        查看升级详情：
            kubectl version
            kubelet --version
    3）确认以上都没有问题，重启kubelet
        sudo systemctl daemon-reload
        sudo systemctl restart kubelet
    4)观察master节点的version版本已经更新到了新的版本
8、升级worker节点
    工作节点上的升级过程应该一次执行一个节点，或者一次执行几个节点， 以不影响运行工作负载所需的最小容量
    1）腾空需要升级的worker节点
    2）升级kubeadm，kubelet，kubectl
    如果机器有外网：
        yum install -y kubeadm-1.20.15-0 kubelet-1.20.15-0 kubectl-1.20.15-0  --disableexcludes=kubernetes
        kubeadm upgrade node
        更新kubelet配置：
            kubeadm upgrade node
        重启kubelet：
            systemctl daemon-reload && systemctl restart kubelet

    如果机器没有外网：
        下载依赖包（备注：需要注意下载同版本的kubelet及kubectl），参考文档：https://blog.csdn.net/m0_60356178/article/details/126784397：
        yum -y install yum-utils
        repotrack --download_path=/opt/package/rpm/ kubeadm-1.20.15-0 
        删除目录/opt/package/rpm/中高版本的kubelet和kubectl的rpm包
        yumdownloader --resolve --destdir /opt/package/rpm/ kubelet-1.20.15-0
        yumdownloader --resolve --destdir /opt/package/rpm/ kubectl-1.20.15-0
        将下载的rpm包copy到需要升级的worker节点，直接安装rpm包，安装命令如下：
        yum -y install *.rpm --obsoletes
        更新kubelet配置：
            kubeadm upgrade node
        重启kubelet：
            systemctl daemon-reload && systemctl restart kubelet

9、检查所有节点的version是否全部升级成功：
        kubectl get nodes