k8s中使用基于nfs的storageclass

Gitlab地址：https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client

1．搭建nfs服务

(1)nfs服务端的安装配置

# 安装nfs
~]# yum -y install nfs-utils rpcbind
# 创建nfs共享目录及设置权限
~]# mkdir /data/k8sdata -p
~]# chmod 755 /data/k8sdata -R
# 配置nfs
~]# cat /etc/exports
/data/k8sdata *(rw,sync,no_root_squash)

nfs主要配置说明：

      * ：表示任何⼈都有权限连接，这儿也可以配置为⼀个⽹段，⼀个 IP，或者域名

      rw：权限为读写

      sync：表示⽂件同时写⼊硬盘和内存

       no_root_squash：当登录 NFS 主机使⽤共享⽬录的使⽤者是 root 时，其权限将被转换成为匿名使⽤者，通常它的 UID 与 GID，都会变成 nobody 身份

# 启动nfs，需要先启动rpcbind再启动nfs
~]# systemctl start rpcbind
~]# systemctl enable rpcbind
~]# systemctl status rpcbind
● rpcbind.service - RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-08-09 14:53:03 CST; 50s ago
 Main PID: 15672 (rpcbind)
   CGroup: /system.slice/rpcbind.service
           └─15672 /sbin/rpcbind -w
~]# systemctl start nfs
~]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
~]# systemctl status nfs
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Sun 2020-08-09 14:57:22 CST; 17s ago
 Main PID: 16369 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service
# 查看具体的挂载权限
~]# cat /var/lib/nfs/etab 
/data/k8sdata *(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=65534,anongid=65534,sec=sys,rw,secure,no_root_squash,no_all_squash)

(2)nfs客户端的安装配置

      Nfs客户端需要安装在使用nfs的各个节点上，即k8s集群的各个node节点上。

# 在客户端上安装启动nfs
~]# yum -y install nfs-utils rpcbind
~]# systemctl start rpcbind
~]# systemctl enable rpcbind
~]# systemctl start nfs
~]# systemctl enable nfs
# 查看nfs共享
~]# showmount -e 192.168.16.133
Export list for 192.168.16.133:
/data/k8sdata *

2、创建基于nfs的storageclass

(1)storageclass

      要使⽤StorageClass，我们就得安装对应的⾃动配置程序，⽐如我们这⾥存储后端使⽤的是 nfs，那么我们就需要使⽤到⼀个 nfs-client的⾃动配置程序，我们也叫它 Provisioner，这个程序帮我们在配置好的nfs 服务器上⾃动创建持久卷，也就是⾃动帮我们创建PV。⾃动创建的PV以${namespace}-${pvcName}-${pvName}这样的命名格式创建在NFS服务器上的共享数据⽬录中，⽽当这个PV被回收后会以archieved-${namespace}-${pvcName}-${pvName}这样的命名格式存在NFS 服务器上。

(2)、安装配置nfs storageclass

# 拉取github相关的资源至本地
~]# git clone https://github.com/kubernetes-retired/external-storage.git
~]# cd external-storage/tree/master/nfs-client/deploy/
# 将deployment.yaml配置文件中的nfs服务器和路径修改为自己nfs服务器和路径
]# cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.16.133
            - name: NFS_PATH
              value: /data/k8sdata
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.16.133
            path: /data/k8sdata
# 修改完成后创建该deployment控制器
deploy]# kubectl apply -f deployment.yaml 
deployment.apps/nfs-client-provisioner created
# 创建sa，并绑定相应的权限
deploy]# kubectl apply -f rbac.yaml 
serviceaccount/nfs-client-provisioner created
# 创建storageclass
deploy]# kubectl apply -f class.yaml 
storageclass.storage.k8s.io/managed-nfs-storage created
# 查看创建的资源
deploy]# kubectl get pods
NAME                                      READY   STATUS             RESTARTS   AGE
nfs-client-provisioner-7d4d65c7b9-l487r   1/1     Running            0          8s
deploy]# kubectl get storageclass
NAME                  PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   fuseim.pri/ifs   Delete          Immediate           false                  22h

3、使用基于nfs的storageclass

（1）创建pvc调用storageclass动态提供pv

      创建了storageclass后，创建pvc之前不需要创建相关的pv,只需调用storageclass提供pv的动态供给。

# 要创建的pvc内容如下
deploy]# cat test-claim.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
# 创建pvc
deploy]# kubectl apply -f test-claim.yaml 
persistentvolumeclaim/test-claim created
deploy]# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-claim   Bound    pvc-15531f5b-b439-43cc-82a8-7152a5ffd50f   1Mi        RWX            managed-nfs-storage   5s

（2）在pod控制器中使用storageclass

      在实际⼯作中，使⽤ StorageClass 更多的是StatefulSet控制器管理的pod，StatefulSet控制器中我们也可以通过⼀个volumeClaimTemplates属性直接使⽤ StorageClass。实际上volumeClaimTemplates下⾯就是⼀个PVC对象的模板，类似于StatefulSet下⾯的template，实际上就是⼀个 Pod 的模板，我们用这种模板就可以动态的去创建pvc对象了。

# statefulset控制器内容
]# cat statefulset-nfs-storageclass.yaml 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nfs-test
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nfs-test
  template:
    metadata:
      labels:
        app: nfs-test
    spec:
      terminationGracePeriodSeconds: 5
      containers:
      - name: nginx
        image: nginx:1.18.0
        ports:
        - name: http
          containerPort: 80
        volumeMounts:
        - name: web
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: web
      annotations:
        volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi
# 创建
]# kubectl apply -f statefulset-nfs-storageclass.yaml
# 查看创建的pvc
]# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
web-nfs-test-0   Bound    pvc-f956a6c8-5382-4a0a-a7ed-d19ac07d83c2   1Gi        RWO            managed-nfs-storage   2m2s
web-nfs-test-1   Bound    pvc-2a25a995-37db-4173-be1e-7a94ea4f9e8f   1Gi        RWO            managed-nfs-storage   119s
# 创建pvc后会在nfs上生成相关的文件
~]# ll /data/k8sdata/
total 0
drwxrwxrwx 2 root root 6 Aug 11 22:06 default-test-claim-pvc-15531f5b-b439-43cc-82a8-7152a5ffd50f
drwxrwxrwx 2 root root 6 Aug 11 22:59 default-web-nfs-test-0-pvc-f956a6c8-5382-4a0a-a7ed-d19ac07d83c2
drwxrwxrwx 2 root root 6 Aug 11 22:59 default-web-nfs-test-1-pvc-2a25a995-37db-4173-be1e-7a94ea4f9e8f