K8S动态PV供给
Kubernetes支持非常丰富的存储卷类型,包括本地存储(节点)和网络存储系统中的诸多存储机制,甚至还支持Secret和ConfigMap这样的特殊存储资源。Kubernetes还设计了一种集群级别的资源PersistentVolume(简称PV),它借由管理员配置存储系统,而后由用户通过“persistentVolumeClaim”(简称PVC)存储卷直接申请使用。Kubernetes的PersistentVolume子系统在用户与管理员之间添加了一个抽象层,从而使得存储系统的使用和管理职能互相解耦。
用户用到持久性存储时 ,需要通过创PVC来绑定匹配的PV,此类操作需求量较大, 或者当管理员手动创建的PV无法满足PVC的所有需求时,系统按PVC的需求标准动态创建适配的PV会为存储管理带来极大的灵活性 。
存储类( storage class )是Kubernetes资源类型的一种,它是由管理员为管理PV之便而按需创建的类别(逻辑组),例如可按存储系统的性能高低分类,或者根据其综合服务质量级别进行分类、依照备份策略分类 ,甚至直接按管理员自定义的标准进行分类等。不过,Kubernetes自 身无法理解“类别 ”到底意味着什么 ,它仅仅是将这些当作PV的特性描述。
StorageClass Spec 中的reclaim Policy 字段: 为当前存储类动态创建的 PV 指定回收策略,可用值为 Delete (默认)和 Retain ;不过,那些由管理员手工创建的 PV 的回收策略则取决于它们自 身的定义 。
1、NFS-CLIENT-PROVISIONER 的工作原理
nfs-client-provisioner 主要分为两部分:
1、在 NFS 共享目录建立 Volume
2、在 Kubernetes 创建持久化存储卷 PV 并与 Volume 做关联,StorageClass 负责建立 PVC 并且与 nfs-client-provisioner 通讯,建立 PVC 和 PV 的连接。
此外还需要创建 ServiceAccount 来保障 nfs-client-provisioner 在 Kubernetes 有足够的权限运行。
2、创建存储系统NFS
]# yum –y install nfs-utils rpcbind #任选一台节点(这里选k8s-master)
]# mkdir -p /data /mydata #创建共享目录
]# cat /etc/exports #配置共享路径
/data 192.168.1.0/24(rw,sync,no_root_squash)
/mydata 192.168.1.0/24(rw,sync,no_root_squash)
]# systemctl restart rpcbind;systemctl restart nfs;systemctl enable nfs
其他节点安装nfs客户端,并验证
]# yum -y install nfs-utils
]# showmount -e 192.168.1.10
Export list for 192.168.1.10:
/data 192.168.1.0/24
/mydata 192.168.1.0/24
3、创建 SERVICE ACCOUNT
配置 nfs-client-provisioner 之前,首先要定义一个Service Account 使用Kubernetes 中的RBAC 机制进行授权。有了相应的权限nfs-client-provisioner可以对StorageClass、PersistentVolumeClaim 进行操作(Role + RoleBinding + ClusterRole + ClusterRoleBinding)。
创建 nfs-client-provisioner-role.yaml 文件编辑并保存,内容如下:
]# cat nfs-client-provisioner-role.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner #Service Account名称
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: leader-locking-nfs-client-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: leader-locking-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
]# kubectl apply -f nfs-client-provisioner-role.yaml #创建ServiceAccount角色和权限绑定
4、安裝 NFS CLIENT PROVISIONER
NFS provisioner 负责在 NFS 的目录中以 ${namespace} - $ {pvcName} - $ {pvName} 格式创建目录,然后告诉 PVC 已经完成 PV 的创建,并且设定两者之间绑定关系。
创建 fs-client-provisioner.yaml 文件,编辑并保存,注意修改NFS服务器所在的IP地址(192.168.1.10),以及NFS服务器共享的路径(/data,/mydata),两处都需要修改为实际的NFS服务器和共享目录;PROVISIONER_NAME 可以更改,但要与后面创建的 StorageClass 中设置一致,此处为“nfs-provisioner”和“mynfs-provisioner”:
]# cat nfs-client-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner #serviceAccount名称
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: nfs-provisioner #与下文storageclass中的metadata. provisioner一致
- name: NFS_SERVER
value: 192.168.1.10 #NFS服务器IP地址,如果为域名,需要能解析
- name: NFS_PATH
value: /data #NFS服务器共享目录
volumes:
- name: nfs-client-root
nfs:
server: 192.168.1.10
path: /data
]# cat mynfs-client-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner #serviceAccount名称
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: mynfs-provisioner #与下文storageclass中的metadata. provisioner一致
- name: NFS_SERVER
value: 192.168.1.10 #NFS服务器IP地址,如果为域名,需要能解析
- name: NFS_PATH
value: /mydata #NFS服务器共享目录
volumes:
- name: nfs-client-root
nfs:
server: 192.168.1.10
path: /mydata
]# kubectl apply -f nfs-client-provisioner.yaml -f mynfs-client-provisioner.yaml
]# kubectl get deployments #查看 NFS Client provisioner 部署的状态
NAME READY UP-TO-DATE AVAILABLE AGE
mynfs-client-provisioner 1/1 1 1 20m
nfs-client-provisioner 1/1 1 1 22m
5、创建 STORAGECLASS
创建 StorageClass:
]# cat nfs-storage.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-storage
provisioner: nfs-provisioner #名称与PROVISIONER_NAME一致
reclaimPolicy: Retain #PV 指定回收策略,默认是delete
parameters:
archiveOnDelete: "false"
]# cat mynfs-storage.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mynfs-storage
provisioner: mynfs-provisioner #名称与PROVISIONER_NAME一致
reclaimPolicy: Delete #PV 指定回收策略,默认是delete
parameters:
archiveOnDelete: "false"
]# kubectl apply -f nfs-storage.yaml -f mynfs-storage.yaml #创建storageclass
]# kubectl get storageClass #查看storageClass状态
NAME PROVISIONER AGE
mynfs-storage mynfs-provisioner 19m
nfs-storage nfs-provisioner 19m
6、创建 PVC
创建PVC之前,先查看下共享目录均为空文件
]# ls -ll /data/
total 0
]# ls -ll /mydata/
total 0
创建 PVC文件:
]# cat nfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-pvc1
annotations:
volume.beta.kubernetes.io/storage-class: "nfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
]# cat mynfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mynfs-pvc1
annotations:
volume.beta.kubernetes.io/storage-class: "mynfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
]# kubectl apply -f mynfs-pvc.yaml -f mynfs-pvc.yaml #创建PVC
]# kubectl get pvc #PVC的STATUS为Bound表示建立成功,VOLUME为动态创建的pv
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pvc1 Bound pvc-f334a676-8e89-4dc3-bae5-5f3c80bda349 1Gi RWX nfs-storage 21s
mynfs-pvc1 Bound pvc-1f71b274-686a-4e5d-813b-a8d320aa76e6 2Gi RWX mynfs-storage 21s
此时查看共享目录,自动创建目录格式为
${namespace}-${pvcName}-${pvName}的文件夹
]# ls /data/
default-nfs-pvc1-pvc-4460195b-dbb8-4315-87a8-bc5965ca4f34
]# ls /mydata/
default-mynfs-pvc1-pvc-aa1872d3-f955-4a93-9845-4e43facacd6b
7、PVC使用默认STORAGECLASS
上面的创建PVC时,通过在pvc的.yaml文件中指定StorageClass(metadata.annotations:
volume.beta.kubernetes.io/storage-class: “mynfs-storage”),关联mynfs-storage这个mynfs-storage,如果不指定后端StorageClass,可以采用默认的存储后端,以下提供两种方式:
1、 用 kubectl patch 命令来更新
]# kubectl patch storageclass mynfs-storage -p '{"metadata": {"annotations":{\
"storageclass.kubernetes.io/is-default-class":"true"}}}'
2、 在.yaml文件中定义metadata.annotations字段值
]# cat mynfs-storage.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mynfs-storage
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: mynfs-provisioner
parameters:
archiveOnDelete: "false"
创建PVC时可以不指定后端存储
]# cat mynfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mynfs-pvc1
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
8、删除PVC测试后端PV
现在通过删除PVC测试自动创建的PV
]# kubectl delete -f mynfs-pvc.yaml -f nfs-pvc.yaml #删除PVC
persistentvolumeclaim "mynfs-pvc1" deleted
persistentvolumeclaim "nfs-pvc1" deleted
查看共享目录,/mydata下的目录已被回收,/data下的依然存在
]# ls /data/
default-nfs-pvc1-pvc-4460195b-dbb8-4315-87a8-bc5965ca4f34
[root@k8s-master ~]# ls -ll /mydata/
total 0
9、创建POD使用PVC
PVC创建成功后,即可在pod中使用
创建一个名称为pod-pvc的pod测试动态pv:
]# cat myapp-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-deploy
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
imagePullPolicy: IfNotPresent
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html/
volumes:
- name: html
persistentVolumeClaim:
claimName: nfs-pvc1 #PVC名称
]# kubectl apply -f myapp-deploy.yaml
]# kubectl get pod
NAME READY STATUS RESTARTS AGE
myapp-deploy-5bcf7dfc7f-gkpbq 1/1 Running 0 6s
myapp-deploy-5bcf7dfc7f-nhm5r 1/1 Running 0 6s
myapp-deploy-5bcf7dfc7f-zgx4q 1/1 Running 0 6s
]# kubectl exec -it myapp-deploy-5bcf7dfc7f-gkpbq -- /bin/sh
/ # echo "Hellow Word" > /usr/share/nginx/html/index.html
在NFS服务端查看,创建的文件(此时文件夹与3.6中不一致,是因为执行了3.7的删除PVC)
]# cat /data/default-nfs-pvc1-pvc-7dcd497d-52ac-4bd7-9a26-cba4d45f145e/index.html
Hellow Word