k8s集成ceph-快照回滚测试

初始环境搭建

1. pv和pvc创建完成后， 创建挂载该rbd的pod， 利用官方文档的busybox容器进行测试。

$ vim rbd-pvc-pod1.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    test: rbd-pvc-pod   
  name: ceph-rbd-pv-pod1
spec:
  containers:
  - name: ceph-rbd-pv-busybox
    image: busybox
    command: ["sleep", "60000"]
    volumeMounts:
    - name: ceph-rbd-vol1
      mountPath: /mnt/ceph-rbd-pvc/busybox
      readOnly: false
  volumes:
  - name: ceph-rbd-vol1
    persistentVolumeClaim:
      claimName: ceph-rbd-pv-claim

2. 将容器的/mnt/ceph-rbd-pvc/busybox目录挂载到创建的testclaim所请求的资源中。

$ kubectl apply -f rbd-pvc-pod1.yaml

$ kubectl get pods -o wide
NAME               READY   STATUS    RESTARTS   AGE    IP            NODE         NOMINATED NODE   READINESS GATES
ceph-rbd-pv-pod1   1/1     Running   0          105s   10.244.1.11   k8s-node03   <none>           <none>

3. 在k8s-node3节点中，查看数据信息

$ docker ps

$ docker exec -it fed3f7843c6f df -h | grep /dev/rbd0
/dev/rbd1               975.9M      2.5M    957.4M   0% /mnt/ceph-rbd-pvc/busybox

$ docker exec -it fed3f7843c6f ls /mnt/ceph-rbd-pvc/busybox

$ docker exec -it fed3f7843c6ftouch /mnt/ceph-rbd-pvc/busybox/file{1..100}

snapshot, protect, clone, map测试

1. 快照的基本实验

# 在k8s-node03(挂载容器的节点)
# 当前容器中创建了100个空文件
$ docker exec -it fed3f7843c6f ls /mnt/ceph-rbd-pvc/busybox/
file1    file20   file32   file44   file56   file68   file8    file91
file10   file21   file33   file45   file57   file69   file80   file92
file100  file22   file34   file46   file58   file7    file81   file93
file11   file23   file35   file47   file59   file70   file82   file94
file12   file24   file36   file48   file6    file71   file83   file95
file13   file25   file37   file49   file60   file72   file84   file96
file14   file26   file38   file5    file61   file73   file85   file97
file15   file27   file39   file50   file62   file74   file86   file98
file16   file28   file4    file51   file63   file75   file87   file99
file17   file29   file40   file52   file64   file76   file88
file18   file3    file41   file53   file65   file77   file89
file19   file30   file42   file54   file66   file78   file9
file2    file31   file43   file55   file67   file79   file90

# 在docker的节点中创建快照
$  rbd snap create kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9@snap-kubernetes-pvc

# 将快照设置为保护快照
$ rbd snap protect kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9@snap-kubernetes-pvc

# 创建克隆
$ rbd clone kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9@snap-kubernetes-pvc kube/clone-kubernetes-pvc

# 在k8s-node01节点
# 查看映像
$ rbd list kube
clone-kubernetes-pvc                                                # 克隆出的映像
kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9

# 将克隆的image映射到内核
$ rbd map kube/clone-kubernetes-pvc
/dev/rbd0

# 挂载路径/mnt/kube到/dev/rbd0
$ mount /dev/rbd0 /mnt/kube/

# 查看当前路径下的文件
$ ls /mnt/kube
file1    file12  file16  file2   file23  file27  file30  file34  file38  file41  file45  file49  file52  file56  file6   file63  file67  file70  file74  file78  file81  file85  file89  file92  file96  
file10   file13  file17  file20  file24  file28  file31  file35  file39  file42  file46  file5   file53  file57  file60  file64  file68  file71  file75  file79  file82  file86  file9   file93  file97  
file100  file14  file18  file21  file25  file29  file32  file36  file4   file43  file47  file50  file54  file58  file61  file65  file69  file72  file76  file8   file83  file87  file90  file94  file98
file11   file15  file19  file22  file26  file3   file33  file37  file40  file44  file48  file51  file55  file59  file62  file66  file7   file73  file77  file80  file84  file88  file91  file95  file99

结论： 可完成拷贝

2. 删除存在克隆的快照的实验

# 移出快照实验，由于克隆出来的映像仍然保留了对父快照的引用，所以若要删除快照需要删除子克隆对父快照的引用。

# 查看当前的image信息
$ rbd list kube
clone-kubernetes-pvc
kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9

# 查看kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9中的快照信息
$ rbd snap list kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9
SNAPID NAME                 SIZE TIMESTAMP
4 snap-kubernetes-pvc 1 GiB Thu Apr 23 18:57:16 2020

# kube/clone-kubernetes-pvc 中存在父快照kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9, 为删除父快照需要拍平克隆
$ rbd flatten kube/clone-kubernetes-pvc
Image flatten: 100% complete...done.

# 解决父快照的protect的状态
$ rbd snap unprotect kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9@snap-kubernetes-pvc

# 移出快照
$ rbd snap remove kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9@snap-kubernetes-pvc
Removing snap: 100% complete...done.

# 查看对当前的克隆是否产生影响
# 查看备份节点
$ ls /mnt/kube/
file1    file12  file16  file2   file23  file27  file30  file34  file38  file41  file45  file49  file52  file56  file6   file63  file67  file70  file74  file78  file81  file85  file89  file92  file96  
file10   file13  file17  file20  file24  file28  file31  file35  file39  file42  file46  file5   file53  file57  file60  file64  file68  file71  file75  file79  file82  file86  file9   file93  file97  
file100  file14  file18  file21  file25  file29  file32  file36  file4   file43  file47  file50  file54  file58  file61  file65  file69  file72  file76  file8   file83  file87  file90  file94  file98
file11   file15  file19  file22  file26  file3   file33  file37  file40  file44  file48  file51  file55  file59  file62  file66  file7   file73  file77  file80  file84  file88  file91  file95  file99

# 查看主节点docker中是否产生影响
docker exec -t d2bff9e0fe27 ls /mnt/ceph-rbd-pvc/busybox/
file1    file20   file32   file44   file56   file68   file8    file91
file10   file21   file33   file45   file57   file69   file80   file92
file100  file22   file34   file46   file58   file7    file81   file93
file11   file23   file35   file47   file59   file70   file82   file94
file12   file24   file36   file48   file6    file71   file83   file95
file13   file25   file37   file49   file60   file72   file84   file96
file14   file26   file38   file5    file61   file73   file85   file97
file15   file27   file39   file50   file62   file74   file86   file98
file16   file28   file4    file51   file63   file75   file87   file99
file17   file29   file40   file52   file64   file76   file88   
file18   file3    file41   file53   file65   file77   file89   
file19   file30   file42   file54   file66   file78   file9
file2    file31   file43   file55   file67   file79   file90

# 结论：删除快照对容器中与备份节点中的数据都不会产生影响

3. 回滚实验

# 当前docker中的状态
$ docker exec -t d2bff9e0fe27 ls /mnt/ceph-rbd-pvc/busybox/
file1    file20   file32   file44   file56   file68   file8    file91
file10   file21   file33   file45   file57   file69   file80   file92
file100  file22   file34   file46   file58   file7    file81   file93
file11   file23   file35   file47   file59   file70   file82   file94
file12   file24   file36   file48   file6    file71   file83   file95
file13   file25   file37   file49   file60   file72   file84   file96
file14   file26   file38   file5    file61   file73   file85   file97
file15   file27   file39   file50   file62   file74   file86   file98
file16   file28   file4    file51   file63   file75   file87   file99
file17   file29   file40   file52   file64   file76   file88   
file18   file3    file41   file53   file65   file77   file89   
file19   file30   file42   file54   file66   file78   file9
file2    file31   file43   file55   file67   file79   file90

# 创建当前状态的快照
$ rbd snap create kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9@snap-kubernetes

# 删除/mnt/ceph-rbd-pvc/busybox/下的file99文件
$ docker exec -t d2bff9e0fe27 rm -rf /mnt/ceph-rbd-pvc/busybox/file{90..100}

# 再次查看docker中的状态
$ docker exec -t d2bff9e0fe27 ls /mnt/ceph-rbd-pvc/busybox/
file1   file2   file3   file4   file5   file6   file7   file8   file9
file10  file20  file30  file40  file50  file60  file70  file80  
file11  file21  file31  file41  file51  file61  file71  file81  
file12  file22  file32  file42  file52  file62  file72  file82
file13  file23  file33  file43  file53  file63  file73  file83
file14  file24  file34  file44  file54  file64  file74  file84
file15  file25  file35  file45  file55  file65  file75  file85
file16  file26  file36  file46  file56  file66  file76  file86
file17  file27  file37  file47  file57  file67  file77  file87
file18  file28  file38  file48  file58  file68  file78  file88
file19  file29  file39  file49  file59  file69  file79  file89

# 查看本机的磁盘挂载情况
$ lsblk
NAME                                                                                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                                                   8:0    0   20G  0 disk
├─sda1                                                                                8:1    0  300M  0 part /boot
├─sda2                                                                                8:2    0    2G  0 part [SWAP]
└─sda3                                                                                8:3    0 17.7G  0 part /
sdb                                                                                   8:16   0   50G  0 disk
└─ceph--30906325--140b--4da5--a307--dd4465bf84b6-osd--block--6c6e9ae0--32c6--488a--a395--cfa4c690a5ac
                                                                                    253:1    0   49G  0 lvm
sdc                                                                                   8:32   0   50G  0 disk
└─ceph--3e3cfdb6--d6cb--4be4--961c--32fb178503d3-osd--block--631c567f--c6f2--4bad--8024--40c0ef7ced9b
                                                                                    253:2    0   49G  0 lvm
sdd                                                                                   8:48   0   50G  0 disk
└─ceph--4ece1eb9--d49d--4239--bf3a--697d83da4093-osd--block--2ba1f77c--783a--429f--87b3--199c1380d391
                                                                                    253:0    0   49G  0 lvm
sr0                                                                                  11:0    1 1024M  0 rom
rbd0                                                                                252:0    0    1G  0 disk /var/lib/kubelet/pods/fc708516-24ae-400f-b728-73b470866250/volumes/kubernetes.io~rbd/pvc-4f4313c3-bf1c-443c-bb58-20ba439e9736

# umount 挂载点
$ umount /var/lib/kubelet/pods/fc708516-24ae-400f-b728-73b470866250/volumes/kubernetes.io~rbd/pvc-4f4313c3-bf1c-443c-bb58-20ba439e9736

# 回滚快照
$ rbd snap rollback kube/kubernetes-dynamic-pvc-b8b16104-8528-11ea-ab78-e2c555d2e9c9@snap-kubernetes
Rolling back to snapshot: 100% complete...done.

# 重新挂载挂载目录
$ mount /dev/rbd0 /var/lib/kubelet/pods/fc708516-24ae-400f-b728-73b470866250/volumes/

# 查看文件内容是否恢复
$ ls /var/lib/kubelet/pods/fc708516-24ae-400f-b728-73b470866250/volumes
file1   file13  file17  file20  file24  file28  file31  file35  file39  file42  file46  file5   file53  file57  file60  file64  file68  file71  file75  file79  file82  file86  file9
file10  file14  file18  file21  file25  file29  file32  file36  file4   file43  file47  file50  file54  file58  file61  file65  file69  file72  file76  file8   file83  file87  
file11  file15  file19  file22  file26  file3   file33  file37  file40  file44  file48  file51  file55  file59  file62  file66  file7   file73  file77  file80  file84  file88  kubernetes.io~secret
file12  file16  file2   file23  file27  file30  file34  file38  file41  file45  file49  file52  file56  file6   file63  file67  file70  file74  file78  file81  file85  file89  


# 回滚实验失败， 可能是和k8s有一定关系，没有真正触动k8s产生的image中。