rook-ceph升级

官方地址：https://rook.io/docs/rook/latest/Upgrade/health-verification/
记录版本升级过程，基本参照官网

升级前检查

所有pod版本

ROOK_CLUSTER_NAMESPACE=rook-ceph #默认rook-ceph名称空间
kubectl -n $ROOK_CLUSTER_NAMESPACE get pods

输出所有Pod状态应该是Running 或者 Completed

查询集群的状态

kubectl -n $ROOK_CLUSTER_NAMESPACE exec -it deployment/rook-ceph-tools -- ceph status

  cluster:
    id:     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum m,n,o (age 16h)
    mgr: b(active, since 21h), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 8 osds: 8 up (since 14h), 8 in (since 16h)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 97 pgs
    objects: 238.83k objects, 921 GiB
    usage:   2.7 TiB used, 15 TiB / 17 TiB avail
    pgs:     97 active+clean
 
  io:
    client:   29 KiB/s rd, 11 MiB/s wr, 5 op/s rd, 43 op/s wr

输出 HEALTH_OK 就可以升级

查询pod对应的版本

kubectl -n $ROOK_CLUSTER_NAMESPACE get deployments -o jsonpath='{range .items[*]}{.metadata.name}{"  \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{"  \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'

kubectl -n $ROOK_CLUSTER_NAMESPACE get jobs -o jsonpath='{range .items[*]}{.metadata.name}{"  \tsucceeded: "}{.status.succeeded}{"      \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'

csi-cephfsplugin-provisioner    req/upd/avl: 2/2/2      rook-version=
csi-rbdplugin-provisioner       req/upd/avl: 2/2/2      rook-version=
rook-ceph-crashcollector-sdk8s-node-01          req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-crashcollector-sdk8s-node-02          req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-crashcollector-sdk8s-node-03          req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-crashcollector-sdk8s-node-04          req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-mds-myfs-a    req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-mds-myfs-b    req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-mgr-a         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-mgr-b         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-mon-m         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-mon-n         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-mon-o         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-operator      req/upd/avl: 1/1/1      rook-version=
rook-ceph-osd-0         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-osd-1         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-osd-2         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-osd-3         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-osd-4         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-osd-5         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-osd-6         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-osd-7         req/upd/avl: 1/1/1      rook-version=v1.10.13
rook-ceph-tools         req/upd/avl: 1/1/1      rook-version=
rook-ceph-osd-prepare-sdk8s-node-01     succeeded: 1            rook-version=v1.10.13
rook-ceph-osd-prepare-sdk8s-node-02     succeeded: 1            rook-version=v1.10.13
rook-ceph-osd-prepare-sdk8s-node-03     succeeded: 1            rook-version=v1.10.13
rook-ceph-osd-prepare-sdk8s-node-04     succeeded: 1            rook-version=v1.10.13

根据当前版本确定升级版本，每次可以同大版本升级，或者夸1个大版本升级；本次CEPH版本升级至v.1.11.11

升级ROOK

克隆rook源码

git clone --single-branch --depth=1 --branch v1.11.11 https://github.com/rook/rook.git

设置环境变量

export ROOK_OPERATOR_NAMESPACE=rook-ceph
export ROOK_CLUSTER_NAMESPACE=rook-ceph

更新公共资源和CRDs

cd rook/deploy/examples
kubectl apply -f common.yaml -f crds.yaml

升级Rook Operator

kubectl -n $ROOK_OPERATOR_NAMESPACE set image deploy/rook-ceph-operator rook-ceph-operator=xxxxxxxx:rook/ceph:v1.11.11

这里可以指定私有的仓库

升级Ceph CSI

由于我是私有仓库，需要手动修改配置升级Ceph CSI

kubectl -n $ROOK_OPERATOR_NAMESPACE edit configmap rook-ceph-operator-config

按照升级说明修改镜像版本

cat operator.yaml |grep IMAGE

  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.8.0"
  # ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.7.0"
  # ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.7.0"
  # ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v3.4.0"
  # ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1"
  # ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v4.1.0"
  # ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.5.0"

等待所有POD升级完成

watch --exec kubectl -n $ROOK_CLUSTER_NAMESPACE get deployments -l rook_cluster=$ROOK_CLUSTER_NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"  \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{"  \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'

Every 2.0s: kubectl -n rook-ceph get deployments -l rook_cluster=rook-ceph -o jsonpath={range .items[*]}{.metadata.name}{"  \treq/upd/avl: "}{.spec.replica...  sdk8s-node-01: Sat Aug 24 08:51:35 2024

rook-ceph-crashcollector-sdk8s-node-01          req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-crashcollector-sdk8s-node-02          req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-crashcollector-sdk8s-node-03          req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-crashcollector-sdk8s-node-04          req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-mds-myfs-a    req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-mds-myfs-b    req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-mgr-a         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-mgr-b         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-mon-m         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-mon-n         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-mon-o         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-osd-0         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-osd-1         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-osd-2         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-osd-3         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-osd-4         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-osd-5         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-osd-6         req/upd/avl: 1/1/1      rook-version=v1.11.11
rook-ceph-osd-7         req/upd/avl: 1/1/1      rook-version=v1.11.11

检查Ceph CSI是否升级完成

kubectl --namespace rook-ceph get pod -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}' -l 'app in (csi-rbdplugin,csi-rbdplugin-provisioner,csi-cephfsplugin,csi-cephfsplugin-provisioner)' | sort | uniq

quay.io/cephcsi/cephcsi:v3.8.0
registry.k8s.io/sig-storage/csi-attacher:v4.1.0
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.7.0
registry.k8s.io/sig-storage/csi-provisioner:v3.4.0
registry.k8s.io/sig-storage/csi-resizer:v1.7.0
registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1

升级CEPH版本