eks使用 external-snapshotter 控制器为ebs卷创建快照和从快照恢复卷

参考资料

• https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/managing-ebs-csi.html

• https://github.com/kubernetes-csi/external-snapshotter

• https://aws.amazon.com/cn/blogs/china/backup-and-restore-ebs-pv-volumes-using-k8s-snapshot-on-amazon-eks/

• https://aws.amazon.com/cn/blogs/containers/using-amazon-ebs-snapshots-for-persistent-storage-with-your-amazon-eks-cluster-by-leveraging-add-ons/

故障描述

在eks集群中创建ebs pvc时，虽然pvc和pv卷能够成功创建，但是ebs csi controller出现以下错误

csi-snapshotter W0403 07:15:53.263394  1 reflector.go:424] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io)                         
csi-snapshotter E0403 07:15:53.263424  1 reflector.go:140] k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1.VolumeSnapshotClass: failed to list *v1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) 

查询github发现同样问题的issue

https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1447

To use the snapshot functionality of the Amazon EBS CSI driver, you must install the external snapshotter before the installation of the add-on…

查看官方文档，有以下描述

要使用 Amazon EBS CSI 驱动程序的快照功能，必须在安装附加组件之前安装外部快照程序。必须按以下顺序安装外部快照程序组件：

• CustomResourceDefinition: volumesnapshotclasses, volumesnapshots and volumesnapshotcontents
• RBAC: ClusterRole, ClusterRoleBinding, etc.
• Controller deployment

卷快照允许您在特定的时间点创建 Amazon EBS 卷的副本。您可以使用此副本将卷恢复到以前的状态或提供新卷

安装external-snapshotter

https://github.com/kubernetes-csi/external-snapshotter

clone仓库

git clone https://github.com/kubernetes-csi/external-snapshotter
cd external-snapshotter

安装snapshot crd

kubectl kustomize client/config/crd | kubectl create -f -

安装Common Snapshot Controller

kubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl create -f -

安装csi driver

kubectl -n kube-system kustomize deploy/kubernetes/snapshot-controller | kubectl create -f -

查看webhook

kubectl get volumesnapshots --selector=snapshot.storage.kubernetes.io/invalid-snapshot-resource: ""
kubectl get volumesnapshotcontents --selector=snapshot.storage.kubernetes.io/invalid-snapshot-content-resource: ""

测试恢复快照

创建存储类和pv

$ cat snapshotclass.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-aws-vsc
driver: ebs.csi.aws.com
deletionPolicy: Delete
kubectl apply -f snapshotclass.yaml

$ cat storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
kubectl apply -f storageclass.yaml

创建pod向pv中写入数据

$ cat claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 4Gi
kubectl apply -f claim.yaml

$ cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: public.ecr.aws/docker/library/centos:centos7.9.2009
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-claim
kubectl apply -f pod.yaml

创建快照

cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: ebs-volume-snapshot
spec:
  volumeSnapshotClassName: csi-aws-vsc
  source:
    persistentVolumeClaimName: ebs-claim
EOF

查看快照

kubectl get volumesnapshot

ebs csi controller日志

csi-snapshotter I0403 07:41:52.848429       1 snapshot_controller.go:291] createSnapshotWrapper: Creating snapshot for content snapcontent-aefef359-170a-49ef-ba34-d5d135aec3b4 through the plugin ...                                                      

控制台查看结果

删除旧卷

kubectl delete -f pod.yaml
kubectl delete -f claim.yaml

从快照中恢复新卷

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-snapshot-restored-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 5Gi
  dataSource:
    name: ebs-volume-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
EOF

恢复应用程序使用新的claim创建卷

$ cat new-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: public.ecr.aws/docker/library/amazonlinux:2
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-snapshot-restored-claim
$ kubectl apply -f new-pod.yaml

新卷创建，并且卷大小改变

查看新卷的信息，时间戳中断，之前的信息留存，成功从快照恢复数据

bash-4.2# cat /data/out.txt
Mon Apr 3 07:41:03 UTC 2023
Mon Apr 3 07:41:08 UTC 2023
Mon Apr 3 07:41:13 UTC 2023
Mon Apr 3 07:41:18 UTC 2023
Mon Apr 3 07:41:23 UTC 2023
Mon Apr 3 07:41:28 UTC 2023
Mon Apr 3 07:41:33 UTC 2023
Mon Apr 3 07:53:02 UTC 2023
Mon Apr 3 07:53:07 UTC 2023
Mon Apr 3 07:53:12 UTC 2023
Mon Apr 3 07:53:17 UTC 2023
Mon Apr 3 07:53:22 UTC 2023
Mon Apr 3 07:53:27 UTC 2023
Mon Apr 3 07:53:32 UTC 2023
Mon Apr 3 07:53:37 UTC 2023
Mon Apr 3 07:53:42 UTC 2023
Mon Apr 3 07:53:47 UTC 2023