kubernetes存储类迁移-备份恢复

背景介绍

kubernetes集群最开始使用了nfs作为存储,随着后续使用过程中数据量逐渐增加,nfs存储性能逐步出现不足,现增加了基于csi的分布式块存储后,需要对原有基于nfs存储类下的pv迁移到新的存储类下。

测试环境

  • k8s集群版本:1.25.12
  • 节点数:4
  • 旧存储类:nfs-storage
  • 新存储类:rook-ceph-block
  • 测试应用:mysql
  • 备份恢复工具:velero
  • 需求:集群管理权限、S3存储

测试方法

  1. 安装minio
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio
  labels:
    app.kubernetes.io/name: minio
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: "minio"
  labels:
    app.kubernetes.io/name: "minio"
spec:
  type: NodePort
  ports:
    - name: tcp
      port: 9000
      targetPort: 9000
    - name: http
      port: 9001
      targetPort: 9001
  selector:
    app.kubernetes.io/name: "minio"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "minio"
  labels:
    app.kubernetes.io/name: "minio"
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: "minio"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: "minio"
    spec:
      containers:
        - name: minio
          image: harbor.gistack.cn/gistack/minio:2023-07-21
          env:
            - name: MINIO_ROOT_USER
              value: minioadmin
            - name: MINIO_ROOT_PASSWORD
              value: minio2023
          args:
            - server
            - /data
            - --console-address
            - :9001
          ports:
            - name: tcp
              containerPort: 9000
              protocol: TCP
            - name: http
              containerPort: 9001
              protocol: TCP
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: minio
  1. 安装velero
[root@rocky8 ~]# wget https://github.com/vmware-tanzu/velero/releases/download/v1.12.2/velero-v1.12.2-linux-amd64.tar.gz
[root@rocky8 ~]# tar xvf velero-v1.12.2-linux-amd64.tar.gz 
velero-v1.12.2-linux-amd64/LICENSE
velero-v1.12.2-linux-amd64/examples/minio/00-minio-deployment.yaml
velero-v1.12.2-linux-amd64/examples/nginx-app/README.md
velero-v1.12.2-linux-amd64/examples/nginx-app/base.yaml
velero-v1.12.2-linux-amd64/examples/nginx-app/with-pv.yaml
velero-v1.12.2-linux-amd64/velero
[root@rocky8 ~]# cd velero-v1.12.2-linux-amd64/
[root@rocky8 velero-v1.12.2-linux-amd64]# ls
LICENSE  examples  velero
[root@rocky8 velero-v1.12.2-linux-amd64]# echo "[default]
> aws_access_key_id = minioadmin
> aws_secret_access_key = minio2023
> " > credentials-minio
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero install \
>   --provider aws \
>   --plugins velero/velero-plugin-for-aws:v1.8.0 \
>   --bucket velero \
>   --secret-file ./credentials-minio \
>   --use-node-agent \
>   --use-volume-snapshots=false \
>   --backup-location-config region=cn-beijing,s3ForcePathStyle="true",s3Url=http://192.168.121.2:32432
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource
CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client
CustomResourceDefinition/backuprepositories.velero.io: created
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
CustomResourceDefinition/datadownloads.velero.io: attempting to create resource
CustomResourceDefinition/datadownloads.velero.io: attempting to create resource client
CustomResourceDefinition/datadownloads.velero.io: created
CustomResourceDefinition/datauploads.velero.io: attempting to create resource
CustomResourceDefinition/datauploads.velero.io: attempting to create resource client
CustomResourceDefinition/datauploads.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
DaemonSet/node-agent: attempting to create resource
DaemonSet/node-agent: attempting to create resource client
DaemonSet/node-agent: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
[root@rocky8 velero-v1.12.2-linux-amd64]# 
  1. 验证velero备份仓库和设置备份策略
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero backup-location get
NAME      PROVIDER   BUCKET/PREFIX   PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        velero          Available   2023-12-29 02:52:26 +0000 UTC   ReadWrite     true
[root@rocky8 velero-v1.12.2-linux-amd64]#cat velero-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  nfs-storage: rook-ceph-block
  1. 安装示例服务mysql
apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: test
  labels:
    app: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  namespace: test
  labels:
    app: mysql
spec:
  storageClassName: nfs-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: test
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - image: mysql:5.7
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: Passw0rd#1
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pv-claim
  1. mysql创建库表并插入数据
mysql> create database test;
Query OK, 1 row affected (0.00 sec)

mysql> use test;
Database changed
mysql> create table newtb (id int);
Query OK, 0 rows affected (0.02 sec)

mysql> insert into newtb (id) values (1);
Query OK, 1 row affected (0.00 sec)

mysql> insert into newtb (id) values (2);
Query OK, 1 row affected (0.00 sec)

mysql> insert into newtb (id) values (3);
Query OK, 1 row affected (0.00 sec)

mysql> select id from newtb;
+------+
| id   |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

mysql> 
  1. 创建备份
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero backup create backup-test --include-namespaces test --include-resources pod,pv,pvc,deploy --default-volumes-to-fs-backup
Backup request "backup-test" submitted successfully.
Run `velero backup describe backup-test` or `velero backup logs backup-test` for more details.
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero backup get 
NAME          STATUS       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
backup-test   InProgress   0        0          2023-12-29 03:47:30 +0000 UTC   29d       default            <none>
[root@rocky8 velero-v1.12.2-linux-amd64]#
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero backup describe backup-test --details
Name:         backup-test
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.25.12
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=25

Phase:  Completed


Namespaces:
  Included:  test
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-12-29 03:47:30 +0000 UTC
Completed:  2023-12-29 03:47:37 +0000 UTC

Expiration:  2024-01-28 03:47:30 +0000 UTC

Total items to be backed up:  21
Items backed up:              21

Resource List:
  apps/v1/Deployment:
    - test/mysql
  apps/v1/ReplicaSet:
    - test/mysql-6d4cc78fd9
  discovery.k8s.io/v1/EndpointSlice:
    - test/mysql-hrqs2
  v1/ConfigMap:
    - test/kube-root-ca.crt
  v1/Endpoints:
    - test/mysql
  v1/Event:
    - test/mysql-6d4cc78fd9-cz54t.17a5306f98ae52c3
    - test/mysql-6d4cc78fd9-cz54t.17a5306ff9652d00
    - test/mysql-6d4cc78fd9-cz54t.17a530701c33cf38
    - test/mysql-6d4cc78fd9-cz54t.17a530701e2834e8
    - test/mysql-6d4cc78fd9-cz54t.17a5307025673793
    - test/mysql-6d4cc78fd9.17a5306f98a6f5fd
    - test/mysql-pv-claim.17a5306fa1d09254
    - test/mysql-pv-claim.17a5306fa209c18c
    - test/mysql-pv-claim.17a5306fa30e0e24
    - test/mysql.17a5306f97c72f22
  v1/Namespace:
    - test
  v1/PersistentVolume:
    - pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac
  v1/PersistentVolumeClaim:
    - test/mysql-pv-claim
  v1/Pod:
    - test/mysql-6d4cc78fd9-cz54t
  v1/Service:
    - test/mysql
  v1/ServiceAccount:
    - test/default

Velero-Native Snapshots: <none included>

kopia Backups:
  Completed:
    test/mysql-6d4cc78fd9-cz54t: mysql-persistent-storage
[root@rocky8 velero-v1.12.2-linux-amd64]# 
  1. 查看minio中数据

image.png
image.png

  1. 修改旧pv的删除策略,保证为Retain
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl get pvc -n test
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-pv-claim   Bound    pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac   1Gi        RWO            nfs-storage    13m
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl edit pv pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac -n test
persistentvolume/pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac edited
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl scale --replicas=0 deploy mysql -n test
deployment.apps/mysql scaled
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl delete pvc mysql-pv-claim -n test
persistentvolumeclaim "mysql-pv-claim" deleted
[root@rocky8 velero-v1.12.2-linux-amd64]#
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl get pv -n test
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                 STORAGECLASS   REASON   AGE
pvc-23a440af-5c6e-4cb5-85b7-964f043b0379   20Gi       RWO            Delete           Bound      default/minio         standard                17m
pvc-54c44a11-77f3-4504-bea4-bb91e3f774ac   1Gi        RWO            Retain           Released   test/mysql-pv-claim   nfs-storage             16m
[root@rocky8 velero-v1.12.2-linux-amd64]# 
  1. 使用velero恢复备份
[root@rocky8 velero-v1.12.2-linux-amd64]# velero backup get 
NAME          STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
backup-test   Completed   0        0          2023-12-29 03:47:30 +0000 UTC   29d       default            <none>
[root@rocky8 velero-v1.12.2-linux-amd64]# velero restore create --from-backup backup-test
Restore request "backup-test-20231229035910" submitted successfully.
Run `velero restore describe backup-test-20231229035910` or `velero restore logs backup-test-20231229035910` for more details.
[root@rocky8 velero-v1.12.2-linux-amd64]# ./velero restore get 
NAME                         BACKUP        STATUS       STARTED                         COMPLETED   ERRORS   WARNINGS   CREATED                         SELECTOR
backup-test-20231229035910   backup-test   InProgress   2023-12-29 03:59:10 +0000 UTC   <nil>       0        0          2023-12-29 03:59:10 +0000 UTC   <none>
[root@rocky8 velero-v1.12.2-linux-amd64]# velero restore get 
NAME                  BACKUP   STATUS      STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
backup-test-20231229035910   backup-test    Completed   2023-12-29 03:59:10 +0000 UTC   2023-12-29 04:06:10 +0000 UTC   0        0          2023-12-29 03:59:10 +0000 UTC   <none>
[root@rocky8 velero-v1.12.2-linux-amd64]#
  1. 验证mysql数据恢复
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl get pvc -n test 
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim   Bound    pvc-dc3f6b8c-53ba-4933-bf46-f8e00a956d7d   1Gi        RWO            rook-ceph-block   9m2s
[root@rocky8 velero-v1.12.2-linux-amd64]#
[root@rocky8 velero-v1.12.2-linux-amd64]# kubectl exec -it -n test deploy/mysql -- bash 
Defaulted container "mysql" out of: mysql, restore-wait (init)
bash-4.2# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.7.44 MySQL Community Server (GPL)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from newtb;
+------+
| id   |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

mysql>
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

小森饭

你的鼓励是我最大的创作动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值