官方地址:https://rook.io/docs/rook/latest/Upgrade/health-verification/
记录版本升级过程,基本参照官网
升级前检查
所有pod版本
ROOK_CLUSTER_NAMESPACE=rook-ceph #默认rook-ceph名称空间
kubectl -n $ROOK_CLUSTER_NAMESPACE get pods
输出所有Pod状态应该是Running 或者 Completed
查询集群的状态
kubectl -n $ROOK_CLUSTER_NAMESPACE exec -it deployment/rook-ceph-tools -- ceph status
cluster:
id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
health: HEALTH_OK
services:
mon: 3 daemons, quorum m,n,o (age 16h)
mgr: b(active, since 21h), standbys: a
mds: 1/1 daemons up, 1 hot standby
osd: 8 osds: 8 up (since 14h), 8 in (since 16h)
data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 238.83k objects, 921 GiB
usage: 2.7 TiB used, 15 TiB / 17 TiB avail
pgs: 97 active+clean
io:
client: 29 KiB/s rd, 11 MiB/s wr, 5 op/s rd, 43 op/s wr
输出 HEALTH_OK 就可以升级
查询pod对应的版本
kubectl -n $ROOK_CLUSTER_NAMESPACE get deployments -o jsonpath='{range .items[*]}{.metadata.name}{" \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{" \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'
kubectl -n $ROOK_CLUSTER_NAMESPACE get jobs -o jsonpath='{range .items[*]}{.metadata.name}{" \tsucceeded: "}{.status.succeeded}{" \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'
csi-cephfsplugin-provisioner req/upd/avl: 2/2/2 rook-version=
csi-rbdplugin-provisioner req/upd/avl: 2/2/2 rook-version=
rook-ceph-crashcollector-sdk8s-node-01 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-crashcollector-sdk8s-node-02 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-crashcollector-sdk8s-node-03 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-crashcollector-sdk8s-node-04 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-mds-myfs-a req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-mds-myfs-b req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-mgr-a req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-mgr-b req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-mon-m req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-mon-n req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-mon-o req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-operator req/upd/avl: 1/1/1 rook-version=
rook-ceph-osd-0 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-osd-1 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-osd-2 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-osd-3 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-osd-4 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-osd-5 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-osd-6 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-osd-7 req/upd/avl: 1/1/1 rook-version=v1.10.13
rook-ceph-tools req/upd/avl: 1/1/1 rook-version=
rook-ceph-osd-prepare-sdk8s-node-01 succeeded: 1 rook-version=v1.10.13
rook-ceph-osd-prepare-sdk8s-node-02 succeeded: 1 rook-version=v1.10.13
rook-ceph-osd-prepare-sdk8s-node-03 succeeded: 1 rook-version=v1.10.13
rook-ceph-osd-prepare-sdk8s-node-04 succeeded: 1 rook-version=v1.10.13
根据当前版本确定升级版本,每次可以同大版本升级,或者夸1个大版本升级;本次CEPH版本升级至v.1.11.11
升级ROOK
克隆rook源码
git clone --single-branch --depth=1 --branch v1.11.11 https://github.com/rook/rook.git
设置环境变量
export ROOK_OPERATOR_NAMESPACE=rook-ceph
export ROOK_CLUSTER_NAMESPACE=rook-ceph
更新公共资源和CRDs
cd rook/deploy/examples
kubectl apply -f common.yaml -f crds.yaml
升级Rook Operator
kubectl -n $ROOK_OPERATOR_NAMESPACE set image deploy/rook-ceph-operator rook-ceph-operator=xxxxxxxx:rook/ceph:v1.11.11
这里可以指定私有的仓库
升级Ceph CSI
由于我是私有仓库,需要手动修改配置升级Ceph CSI
kubectl -n $ROOK_OPERATOR_NAMESPACE edit configmap rook-ceph-operator-config
按照升级说明修改镜像版本
cat operator.yaml |grep IMAGE
# ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.8.0"
# ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.7.0"
# ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.7.0"
# ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v3.4.0"
# ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1"
# ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v4.1.0"
# ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.5.0"
等待所有POD升级完成
watch --exec kubectl -n $ROOK_CLUSTER_NAMESPACE get deployments -l rook_cluster=$ROOK_CLUSTER_NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{" \treq/upd/avl: "}{.spec.replicas}{"/"}{.status.updatedReplicas}{"/"}{.status.readyReplicas}{" \trook-version="}{.metadata.labels.rook-version}{"\n"}{end}'
Every 2.0s: kubectl -n rook-ceph get deployments -l rook_cluster=rook-ceph -o jsonpath={range .items[*]}{.metadata.name}{" \treq/upd/avl: "}{.spec.replica... sdk8s-node-01: Sat Aug 24 08:51:35 2024
rook-ceph-crashcollector-sdk8s-node-01 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-crashcollector-sdk8s-node-02 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-crashcollector-sdk8s-node-03 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-crashcollector-sdk8s-node-04 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-mds-myfs-a req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-mds-myfs-b req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-mgr-a req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-mgr-b req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-mon-m req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-mon-n req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-mon-o req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-osd-0 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-osd-1 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-osd-2 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-osd-3 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-osd-4 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-osd-5 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-osd-6 req/upd/avl: 1/1/1 rook-version=v1.11.11
rook-ceph-osd-7 req/upd/avl: 1/1/1 rook-version=v1.11.11
检查Ceph CSI是否升级完成
kubectl --namespace rook-ceph get pod -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}' -l 'app in (csi-rbdplugin,csi-rbdplugin-provisioner,csi-cephfsplugin,csi-cephfsplugin-provisioner)' | sort | uniq
quay.io/cephcsi/cephcsi:v3.8.0
registry.k8s.io/sig-storage/csi-attacher:v4.1.0
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.7.0
registry.k8s.io/sig-storage/csi-provisioner:v3.4.0
registry.k8s.io/sig-storage/csi-resizer:v1.7.0
registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1