NFS PV自动供给

一 NFS基本使用

1.1 服务端

• 安装NFS，nfs必须在rpcbind后面启动，因为要注册nfs的信息到rpcbind服务中
• 改完配置需要重启nfs或者重新加载nfs配置文件，配置文件默认为 /etc/exports

### 离线环境下，nfs-utils和rpcbind都可以使用yum install  --downloadonly --downloaddir=路径  把rpm包下载到本地目录
### 然后使用rpm -ivh --force --nodeps *.rpm安装nfs-utils、rpcbind
# yum install nfs-utils
# yum install rpcbind
# systemctl start rpcbind && systemctl enable rpcbind
# systemctl start nfs && systemctl enable nfs

• 创建共享目录，并把共享目录加入到nfs配置中
• rw(读写权限)、ro(只读)、sync(客户端服务端数据同步强一致性)，no_root_squash(服务端文件组用户root)

# mkdir /k8s
# vi /etc/exports
/k8s 192.168.217.0/24(rw,sync,no_root_squash)
# systemctl reload nfs

至此，服务端nfs安装配置工作已经完成，下面需要用客户端使用共享目录(/k8s)

1.2 客户端

• 只需安装nfs、rpcbind，不需要启动

# yum install -y nfs rpcbind

• 挂载服务端的/k8s目录到本地（测试nfs是否正常运行）

# mkdir /mnt/k8s
# mount -t nfs 192.168.217.200:/k8s /mnt/k8s
# echo "hello nfs" > /mnt/k8s/hello.txt

• 查看服务端是否有hello.txt

# ll /k8s/  # root root由no_root_squash属性决定的
总用量 4
-rw-r--r--. 1 root root 2 5月  10 12:47 hello.txt

到此，nfs服务器已经正常工作了。

二 NFS自动供给

2.1 什么是pv自动供给？

当集群中有上万个pv的时候，挨个创建会十分麻烦，于是乎有了一种可以动态创建pv的方式，这便是pv动态供给。pv动态供给需要nfs-client-provisioner这个deployment绑定一个nfs的pv共用目录。然后通过storageClass来自动分配nfs存储。

2.2helm安装NFS自动供给

• 目录结构

# tree .
.
├── charts
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── rbac.yaml
│   └── storage.yaml
└── values.yaml

• rbac.yaml（不在default ns时，需要手动添加权限）

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner
  namespace: {{ .Release.Namespace }}
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-runner
  namespace: {{ .Release.Namespace }}
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get","create","list", "watch","update"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: {{ .Release.Namespace }}
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

• deployment.yaml 使用vbouchaud/nfs-client-provisioner:latest代替google源

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
  namespace: {{ .Release.Namespace }}
  labels:
    name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      name: nfs-client-provisioner
  template:
    metadata:
      labels:
        name: nfs-client-provisioner
    spec:
      nodeSelector:
       kubernetes.io/hostname: k8s-master
      serviceAccount: nfs-provisioner
      containers:
        - name: nfs-client-provisioner
          image: vbouchaud/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.217.200
            - name: NFS_PATH
              value: /k8s
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.217.200
            path: /k8s

• storage.yaml指定存储类，provisioner需要和上面的一致(绑定deployment)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: grafana-nfs
  namespace: {{ .Release.Namespace }}
provisioner: fuseim.pri/ifs
reclaimPolicy: Retain

provisioner: 该字段指定使用存储卷类型，不同的存储卷提供者类型这里要修改成对应的值。
reclaimPolicy：有两种策略：Delete、Retain。默认是Delet
用户删除PVC释放对PV的占用后，系统根据PV的"reclaim policy"决定对PV执行何种回收操作。 目前，"reclaim policy"有三种方式：Retained、Recycled、Deleted。推荐使用Retained，pv改为Released状态、数据卷进行保留。

• 安装

# kubectl create ns nfs
# helm install nfs . -n nfs

2.3 动态pv测试

使用nginx statefulset 测试动态pv

• kubectl apply -f nginx.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: nginx:1.17
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "grafana-nfs"
      resources:
        requests:
          storage: 1Gi

• 查看三个pod的运行状态

# kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
web-0   1/1     Running   0          15m   10.244.169.144   k8s-node2    <none>           <none>
web-1   1/1     Running   0          15m   10.244.36.68     k8s-node1    <none>           <none>
web-2   1/1     Running   0          15m   10.244.235.203   k8s-master   <none>           <none>

• 查看通过grafana-nfs这个存储类创建的pvc的状态，全部绑定中。

# kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-e4255760-824d-437f-b6db-0417970207a0   1Gi        RWO            grafana-nfs    17m
www-web-1   Bound    pvc-8b7fa2e3-c0e6-4a3f-9516-9b28f554bbde   1Gi        RWO            grafana-nfs    17m
www-web-2   Bound    pvc-df1c4198-d955-4c3f-b101-d74c99281454   1Gi        RWO            grafana-nfs    17m

• 当三个pod都成功运行之后，可以在nfs这台的服务器下面看到，自动生成的pv持久卷

# ls /k8s/
default-www-web-0-pvc-e4255760-824d-437f-b6db-0417970207a0  default-www-web-2-pvc-df1c4198-d955-4c3f-b101-d74c99281454
default-www-web-1-pvc-8b7fa2e3-c0e6-4a3f-9516-9b28f554bbde

到此为止，通过nfs实现pv自动供给的功能就完成了。