kubernetes部署nfs-subdir-external-provisioner

nfs-subdir-external-provisioner简介

nfs-subdir-external-provisioner可动态为kubernetes提供pv卷，是Kubernetes的简易NFS的外部provisioner，本身不提供NFS，需要现有的NFS服务器提供存储。持久卷目录的命名规则为: ${namespace}-${pvcName}-${pvName}。

项目地址：https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner

搭建NFS服务器

集群外准备一个节点10.0.0.5作为NFS server服务器，这里作为测试，使用docker部署NFS服务器：

docker run -d --name nfs-server \
    --privileged \
    --restart always \
    -p 2049:2049 \
    -v /nfs-share:/nfs-share \
    -e SHARED_DIRECTORY=/nfs-share \
    itsthenetwork/nfs-server-alpine:latest

或者手动方式部署nfs服务器：

yum -y install nfs-utils

mkdir -p /nfs-share
chmod -R 777 /nfs-share

cat >> /etc/exports << EOF
/nfs-share *(rw,no_root_squash,sync)
EOF

exportfs -arv
exportfs -s

systemctl enable --now rpcbind
systemctl enable --now nfs-server

kubernetes集群所有节点需要安装nfs客户端

yum install -y nfs-utils

作为准备工作，我们已经搭建了一个 NFS 服务器，共享目录为 /nfs-share。下面部署nfs-subdir-external-provisioner。

helm部署nfs-subdir-external-provisioner

helm hub地址：https://artifacthub.io/packages/helm/nfs-subdir-external-provisioner/nfs-subdir-external-provisioner

添加helm仓库

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

使用helm部署，镜像问题需自行解决，这里使用dockerhub搜索到的镜像：

helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --namespace=nfs-provisioner \
  --create-namespace \
  --set image.repository=willdockerhub/nfs-subdir-external-provisioner \
  --set image.tag=v4.0.2 \
  --set replicaCount=2 \
  --set storageClass.name=nfs-client \
  --set storageClass.defaultClass=true \
  --set nfs.server=10.0.0.5 \
  --set nfs.path=/

注意：如果nfs server使用docker部署，在nfsv4中nfs.path直接使用/即可，手动部署nfs server时nfs.path指定为/nfs-share。

查看创建的pods

[root@master ~]# kubectl -n nfs-provisioner get pods
NAME                                               READY   STATUS    RESTARTS   AGE
nfs-subdir-external-provisioner-855665bf6c-c7g4f   1/1     Running   0          19m
nfs-subdir-external-provisioner-855665bf6c-fn67g   1/1     Running   0          164m

查看创建的storageclass

[root@master ~]# kubectl get sc
NAME                   PROVISIONER                                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-subdir-external-provisioner   Delete          Immediate           true                   166m

从NFS动态申请PV

示例1：创建一个wordpress应用进行验证，其中wordpress以及mariadb引用了持久化存储

helm repo add bitnami https://charts.bitnami.com/bitnami

helm install wordpress bitnami/wordpress \
  --namespace=wordpress \
  --create-namespace \
  --set global.storageClass=nfs-client \
  --set wordpressUsername=admin \
  --set wordpressPassword=password \
  --set replicaCount=2 \
  --set service.type=NodePort \
  --set service.nodePorts.http=30808 \
  --set persistence.enabled=true \
  --set persistence.size=10Gi \
  --set volumePermissions.enabled=true \
  --set mariadb.enabled=true \
  --set mariadb.architecture=standalone \
  --set mariadb.auth.rootPassword=password \
  --set mariadb.auth.password=password \
  --set mariadb.primary.persistence.enabled=true \
  --set mariadb.primary.persistence.size=8Gi \
  --set memcached.enabled=true \
  --set wordpressConfigureCache=true

查看创建的pod及service:

[root@master ~]# kubectl -n wordpress get pods
NAME                                   READY   STATUS    RESTARTS   AGE
wordpress-798966ff57-8jfgb             1/1     Running   1          5m20s
wordpress-798966ff57-hk6gz             1/1     Running   0          5m20s
wordpress-mariadb-0                    1/1     Running   0          5m20s
wordpress-memcached-67744fbd98-v8j8z   1/1     Running   0          5m20s

[root@master ~]# kubectl -n wordpress get svc
NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
wordpress             ClusterIP   10.233.63.20   <none>        80/TCP,443/TCP   3m53s
wordpress-mariadb     ClusterIP   10.233.21.68   <none>        3306/TCP         3m53s
wordpress-memcached   ClusterIP   10.233.4.89    <none>        11211/TCP        3m53s

查看自动创建的pvc及pv

[root@master ~]# kubectl -n wordpress get pvc
NAME                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-wordpress-mariadb-0   Bound    pvc-a677fa75-fdb0-4a3c-aa7e-de0e9be3baac   8Gi        RWO            nfs-client     141m
wordpress                  Bound    pvc-c5b58ca0-73e3-4ef9-9113-9344524776c2   10Gi       RWO            nfs-client     6m40s

[root@master ~]# kubectl -n wordpress get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                STORAGECLASS   REASON   AGE
pvc-a677fa75-fdb0-4a3c-aa7e-de0e9be3baac   8Gi        RWO            Delete           Bound    wordpress/data-wordpress-mariadb-0   nfs-client              141m
pvc-c5b58ca0-73e3-4ef9-9113-9344524776c2   10Gi       RWO            Delete           Bound    wordpress/wordpress                  nfs-client              6m42s

连接到NFS server查看本地数据

[root@nfs-server ~]# ll /nfs-share
total 24
drwxrwxrwx 3 root root 4096 Aug 16 14:23 wordpress-data-wordpress-mariadb-0-pvc-a54d15ff-fec7-4b0a-9c60-3181926409a7
drwxrwxrwx 3 root root 4096 Aug 16 13:58 wordpress-wordpress-pvc-cb951124-aec5-4650-8f1d-2a4850f5d34d

浏览器访问wordpress应用

示例2：创建一个tomcat应用进行验证

helm repo add bitnami https://charts.bitnami.com/bitnami

helm install tomcat bitnami/tomcat \
  --namespace=apps \
  --create-namespace \
  --set global.storageClass=nfs-client \
  --set replicaCount=2 \
  --set service.type=NodePort \
  --set service.nodePort=30809

查看创建的相关资源：

[root@master ~]# kubectl -n apps get pods
NAME                      READY   STATUS    RESTARTS   AGE
tomcat-776b4846fd-7m8p2   1/1     Running   0          44m
tomcat-776b4846fd-w4vms   1/1     Running   0          2m54s

[root@master ~]# kubectl -n apps get svc
NAME     TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
tomcat   NodePort   10.233.50.18   <none>        80:30809/TCP   44m

[root@master ~]# kubectl -n apps get pvc
NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
tomcat   Bound    pvc-6c26f444-92f0-45bc-9884-b3ff87bbb338   8Gi        RWO            nfs-client     44m

[root@master ~]# kubectl -n apps get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                STORAGECLASS   REASON   AGE
pvc-6c26f444-92f0-45bc-9884-b3ff87bbb338   8Gi        RWO            Delete           Bound    apps/tomcat                          nfs-client              44m
pvc-a54d15ff-fec7-4b0a-9c60-3181926409a7   8Gi        RWO            Delete           Bound    wordpress/data-wordpress-mariadb-0   nfs-client              53m
pvc-cb951124-aec5-4650-8f1d-2a4850f5d34d   10Gi       RWO            Delete           Bound    wordpress/wordpress                  nfs-client              53m

浏览器访问tomcat: