背景介绍
kubernetes集群最开始使用了nfs作为存储,随着后续使用过程中数据量逐渐增加,nfs存储性能逐步出现不足,现增加了基于csi的分布式块存储后,需要对原有基于nfs存储类下的pv迁移到新的存储类下。
测试环境
- k8s集群版本:1.25.12
- 节点数:4
- 旧存储类:nfs-storage
- 新存储类:rook-ceph-block
- 测试应用:mysql
- 备份恢复工具:velero
- 需求:集群管理权限、S3存储
测试方法
- 安装minio
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minio
labels:
app.kubernetes.io/name: minio
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
name: "minio"
labels:
app.kubernetes.io/name: "minio"
spec:
type: NodePort
ports:
- name: tcp
port: 9000
targetPort: 9000
- name: http
port: 9001
targetPort: 9001
selector:
app.kubernetes.io/name: "minio"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: "minio"
labels:
app.kubernetes.io/name: "minio"
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: "minio"
template:
metadata:
labels:
app.kubernetes.io/name: "minio"
spec:
containers:
- name: minio
image: harbor.gistack.cn/gistack/minio:2023-07-21
env:
- name: MINIO_ROOT_USER
value: minioadmin
- name: MINIO_ROOT_PASSWORD
value: minio2023
args:
- server
- /data
- --console-address
- :9001
ports:
- name: tcp
containerPort: 9000
protocol: TCP
- name: http
containerPort: 9001
protocol: TCP
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: minio
- 安装velero
[root@rocky8 ~]# wget https://github.com/vmware-tanzu/velero/releases/download/v1.12.2/velero-v1.12.2-linux-amd64.tar.gz
[root@rocky8 ~]# tar xvf velero-v1.12.2-linux-amd64.tar.gz
velero-v1.12.2-linux-amd64/LICENSE
velero-v1.12.2-linux-amd64/examples/minio/00-minio-deployment.yaml
velero-v1.12.2-linux-amd64/examples/nginx-app/README.md
velero-v1.12.2-linux-amd64/examples/nginx-app/base.yaml
velero-v1.12.2-linux-amd64/examples/nginx-app/with-pv.yaml
velero-v1.12.2-linux-amd64/velero
[root@rocky8 ~]# cd velero-v1.12.2-linux-amd64/
[root@rocky8 velero-v1.12.2-linux-amd64]# ls
LICENSE examples velero
[root@rocky8 velero-v1.12.2-linux-amd64]# echo "[default]
> aws_access_key_id = minioadmin
> aws_secret_access_key = minio2023
> " > credentials-minio
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero install \
> --provider aws \
> --plugins velero/velero-plugin-for-aws:v1.8.0 \
> --bucket velero \
> --secret-file ./credentials-minio \
> --use-node-agent \
> --use-volume-snapshots=false \
> --backup-location-config region=cn-beijing,s3ForcePathStyle="true",s3Url=http://192.168.121.2:32432
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client
CustomResourceDefinition/backuprepositories.velero.io: created
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
CustomResourceDefinition/datadownloads.velero.io: attempting to create resource
CustomResourceDefinition/datadownloads.velero.io: attempting to create resource client
CustomResourceDefinition/datadownloads.velero.io: created
CustomResourceDefinition/datauploads.velero.io: attempting to create resource
CustomResourceDefinition/datauploads.velero.io: attempting to create resource client
CustomResourceDefinition/datauploads.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
DaemonSet/node-agent: attempting to create resource
DaemonSet/node-agent: attempting to create resource client
DaemonSet/node-agent: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
[root@rocky8 velero-v1.12.2-linux-amd64]#
- 验证velero备份仓库和设置备份策略
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
default aws velero Available 2023-12-29 02:52:26 +0000 UTC ReadWrite true
[root@rocky8 velero-v1.12.2-linux-amd64]#cat velero-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: change-storage-class-config
namespace: velero
labels:
velero.io/plugin-config: ""
velero.io/change-storage-class: RestoreItemAction
data:
nfs-storage: rook-ceph-block
- 安装示例服务mysql
apiVersion: v1
kind: Service
metadata:
name: mysql
namespace: test
labels:
app: mysql
spec:
ports:
- port: 3306
selector:
app: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
namespace: test
labels:
app: mysql
spec:
storageClassName: nfs-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
namespace: test
labels:
app: mysql
spec:
selector:
matchLabels:
app: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:5.7
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: Passw0rd#1
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
- mysql创建库表并插入数据
mysql> create database test;
Query OK, 1 row affected (0.00 sec)
mysql> use test;
Database changed
mysql> create table newtb (id int);
Query OK, 0 rows affected (0.02 sec)
mysql> insert into newtb (id) values (1);
Query OK, 1 row affected (0.00 sec)
mysql> insert into newtb (id) values (2);
Query OK, 1 row affected (0.00 sec)
mysql> insert into newtb (id) values (3);
Query OK, 1 row affected (0.00 sec)
mysql> select id from newtb;
+------+
| id |
+------+
| 1 |
| 2 |
| 3 |
+------+
3 rows in set (0.00 sec)
mysql>
- 创建备份
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero backup create backup-test --include-namespaces test --include-resources pod,pv,pvc,deploy --default-volumes-to-fs-backup
Backup request "backup-test" submitted successfully.
Run `velero backup describe backup-test` or `velero backup logs backup-test` for more details.
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
backup-test InProgress 0 0 2023-12-29 03:47:30 +0000 UTC 29d default <none>
[root@rocky8 velero-v1.12.2-linux-amd64]#
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero backup describe backup-test --details
Name: backup-test
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/resource-timeout=10m0s
velero.io/source-cluster-k8s-gitversion=v1.25.12
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=25
Phase: Completed
Namespaces:
Included: test
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Or label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: auto
Snapshot Move Data: false
Data Mover: velero
TTL: 720h0m0s
CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 4h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2023-12-29 03:47:30 +0000 UTC
Completed: 2023-12-29 03:47:37 +0000 UTC
Expiration: 2024-01-28 03:47:30 +0000 UTC
Total items to be backed up: 21
Items backed up: 21
Resource List:
apps/v1/Deployment:
- test/mysql
apps/v1/ReplicaSet:
- test/mysql-6d4cc78fd9
discovery.k8s.io/v1/EndpointSlice:
- test/mysql-hrqs2
v1/ConfigMap:
- test/kube-root-ca.crt
v1/Endpoints:
- test/mysql
v1/Event:
- test/mysql-6d4cc78fd9-cz54t.17a5306f98ae52c3
- test/mysql-6d4cc78fd9-cz54t.17a5306ff9652d00
- test/mysql-6d4cc78fd9-cz54t.17a530701c33cf38
- test/mysql-6d4cc78fd9-cz54t.17a530701e2834e8
- test/mysql-6d4cc78fd9-cz54t.17a5307025673793
- test/mysql-6d4cc78fd9.17a5306f98a6f5fd
- test/mysql-pv-claim.17a5306fa1d09254
- test/mysql-pv-claim.17a5306fa209c18c
- test/mysql-pv-claim.17a5306fa30e0e24
- test/mysql.17a5306f97c72f22
v1/Namespace:
- test
v1/PersistentVolume:
- pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac
v1/PersistentVolumeClaim:
- test/mysql-pv-claim
v1/Pod:
- test/mysql-6d4cc78fd9-cz54t
v1/Service:
- test/mysql
v1/ServiceAccount:
- test/default
Velero-Native Snapshots: <none included>
kopia Backups:
Completed:
test/mysql-6d4cc78fd9-cz54t: mysql-persistent-storage
[root@rocky8 velero-v1.12.2-linux-amd64]#
- 查看minio中数据
- 修改旧pv的删除策略,保证为Retain
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl get pvc -n test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac 1Gi RWO nfs-storage 13m
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl edit pv pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac -n test
persistentvolume/pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac edited
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl scale --replicas=0 deploy mysql -n test
deployment.apps/mysql scaled
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl delete pvc mysql-pv-claim -n test
persistentvolumeclaim "mysql-pv-claim" deleted
[root@rocky8 velero-v1.12.2-linux-amd64]#
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl get pv -n test
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-23a440af-5c6e-4cb5-85b7-964f043b0379 20Gi RWO Delete Bound default/minio standard 17m
pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac 1Gi RWO Retain Released test/mysql-pv-claim nfs-storage 16m
[root@rocky8 velero-v1.12.2-linux-amd64]#
- 使用velero恢复备份
[root@rocky8 velero-v1.12.2-linux-amd64]# velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
backup-test Completed 0 0 2023-12-29 03:47:30 +0000 UTC 29d default <none>
[root@rocky8 velero-v1.12.2-linux-amd64]# velero restore create --from-backup backup-test
Restore request "backup-test-20231229035910" submitted successfully.
Run `velero restore describe backup-test-20231229035910` or `velero restore logs backup-test-20231229035910` for more details.
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero restore get
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
backup-test-20231229035910 backup-test InProgress 2023-12-29 03:59:10 +0000 UTC <nil> 0 0 2023-12-29 03:59:10 +0000 UTC <none>
[root@rocky8 velero-v1.12.2-linux-amd64]# velero restore get
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
backup-test-20231229035910 backup-test Completed 2023-12-29 03:59:10 +0000 UTC 2023-12-29 04:06:10 +0000 UTC 0 0 2023-12-29 03:59:10 +0000 UTC <none>
[root@rocky8 velero-v1.12.2-linux-amd64]#
- 验证mysql数据恢复
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl get pvc -n test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-dc3f6b8c-53ba-4933-bf46-f8e00a956d7d 1Gi RWO rook-ceph-block 9m2s
[root@rocky8 velero-v1.12.2-linux-amd64]#
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl exec -it -n test deploy/mysql -- bash
Defaulted container "mysql" out of: mysql, restore-wait (init)
bash-4.2# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.7.44 MySQL Community Server (GPL)
Copyright (c) 2000, 2023, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from newtb;
+------+
| id |
+------+
| 1 |
| 2 |
| 3 |
+------+
3 rows in set (0.00 sec)
mysql>