kubernetes中部署nfs provisioner，提供基于nfs的StorageClass

kubernetes存储

PVC、PC、StorageClass关系图

存储生命周期

安装NFS并启动服务

执行命令

#master节点安装nfs
yum -y install nfs-utils

NFS_DIR=/nfs/data/

#创建nfs目录
mkdir -p $NFS_DIR

#修改权限
chmod -R 777 $NFS_DIR

#编辑export文件
vim /etc/exports
/nfs/data *(rw,no_root_squash,sync)

#配置生效
exportfs -r
#查看生效
exportfs
# 正常情况会显示  /nfs/data   <world>

#启动rpcbind、nfs服务，注意如果失败，常识重启系统后再试
systemctl restart rpcbind && systemctl enable rpcbind
systemctl restart nfs && systemctl enable nfs

#查看 RPC 服务的注册状况
rpcinfo -p localhost

#showmount测试
showmount -e 192.168.1.2

#所有node节点安装客户端
yum -y install nfs-utils
systemctl start nfs && systemctl enable nfs

K8s中部署NFS

NFS subdir external provisioner 使用现有的的NFS 服务器来支持通过 Persistent Volume Claims 动态供应 Kubernetes Persistent Volumes。持久卷默认被配置为$namespace-${pvcName}-${pvName}，使用这个必须已经拥有 NFS 服务器。

https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
https://blog.csdn.net/networken/article/details/86697018

下载项目，拷贝deploy到服务器

下载镜像，国内镜像地址 dockerhub上搜索 dyrnq/nfs-subdir-external-provisioner:v4.0.2

registry.cn-hangzhou.aliyuncs.com/k8s-image-mirrors/nfs-subdir-external-provisioner:v4.0.2

rbac配置

rbac.yaml如下

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: <YOUR_NFS_NS>
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: <YOUR_NFS_NS>
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: <YOUR_NFS_NS>
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: <YOUR_NFS_NS>
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: <YOUR_NFS_NS>
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

Deployment配置

这里修改的参数包括NFS服务器所在的IP地址（192.168.1.2），以及NFS服务器共享的路径（/nfs/data），两处都需要修改为你实际的NFS服务器和共享目录。另外修改nfs-client-provisioner镜像从dockerhub拉取。
deployment.yaml内容如下，需要替换**<YOUR_NFS_NS>、<YOUR_NFS_SERVER_IP>、<YOUR_NFS_SERVER_SHARE>**

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: <YOUR_NFS_NS>
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: <YOUR_NFS_NS>/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: <YOUR_NFS_SERVER_IP>
            - name: NFS_PATH
              value: <YOUR_NFS_SERVER_SHARE>
      volumes:
        - name: nfs-client-root
          nfs:
            server: <YOUR_NFS_SERVER_IP>
            path: <YOUR_NFS_SERVER_SHARE>

StorageClass配置

storage class的定义，需要注意的是：provisioner属性要等于驱动所传入的环境变量PROVISIONER_NAME的值。否则，驱动不知道知道如何绑定storage class。

class.yaml内容如下

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: <YOUR_NFS_NS>/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

部署

#!/bin/bash
YOUR_NFS_NS=mapgis-cloud
YOUR_NFS_SERVER_IP=192.168.1.2
YOUR_NFS_SERVER_SHARE=/mnt/sdb/nfs-data

cd `dirname $0`
# 模板拷贝到新目录下替换
mkdir ./work
cp *.yaml ./work/
cd work
sed -i "s#<YOUR_NFS_NS>#${YOUR_NFS_NS}#g" deployment.yaml rbac.yaml class.yaml
sed -i "s#<YOUR_NFS_SERVER_IP>#${YOUR_NFS_SERVER_IP}#g" deployment.yaml
sed -i "s#<YOUR_NFS_SERVER_SHARE>#${YOUR_NFS_SERVER_SHARE}#g" deployment.yaml
kubectl apply -f rbac.yaml
kubectl apply -f deployment.yaml
kubectl apply -f class.yaml

创建PVC示例

1. 基于storageClass创建PVC

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  storageClassName: managed-nfs-storage
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

2. 示例pod中使用pvc

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: 192.168.1.2:5000/busybox:stable
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim

PV回收模式

容器本身是无状态的，出现问题后随时可销毁，但保存数据的的卷不能被销毁，需要创建一个持久卷。持久卷删除时有三种回售模式，

保持（Retain）:删除PV后后端存储上的数据仍然存在，如需彻底删除则需要手动删除后端存储volume
删除（Delete）：删除被PVC释放的PV和后端存储volume

默认情况下是delete模式

修改命令

kubectl patch pv pvc-89ecc53b-725c-11ea-a8e2-005056aa2a6d -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

通过 PersistentVolume 和 PersistentVolumeClaim，Kubernetes 分离了提供存储和使用存储着两个关注点：

• PersistentVolumeClaim 必须定义在与应用程序相同的名称空间中，关注应用程序如何使用存储，通常由应用程序管理员或开发人员负责
• PersistentVolume 只能定义在集群层面，关注集群如何提供存储，通常由集群管理员或者运维人员负责