Kubernetes从0到1-第二篇 安装NFS Provisioner存储置备程序

想必大家对PV、PVC的概念都不会陌生，它们的好处也应该了解，对静态卷配置、动态卷配置也应该有了解，所以这里不再做讲解。熟悉ACK的都知道，ACK集群有flexvolume、CSI存储置备程序，现在我们基于NFS服务器搭建一套自己的存储置备程序。

架构

NFS subdir external provisioner 可动态为kubernetes提供pv卷，是Kubernetes的简易NFS的外部provisioner，本身不提供NFS，需要现有的NFS服务器提供存储。持久卷目录的命名规则为: $namespace-${pvcName}-${pvName}。

准备工作

kubectl自动补全

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

安装helm

wget https://get.helm.sh/helm-v3.3.4-linux-amd64.tar.gz
tar -zxvf helm-v3.3.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
helm version

命令补全

vim ~/.bashrc
source <(helm completion bash)
 
source ~/.bashrc

部署NFS服务器

yum -y install nfs-utils rpcbind

mkdir -p /nfs-data
chmod -R 777 /nfs-data

cat >> /etc/exports << EOF
/nfs-data *(rw,no_root_squash,sync)
EOF

exportfs -arv
exportfs -s

# 启动，注意：先启动rpc服务，再启动nfs服务

systemctl start rpcbind
systemctl enable rpcbind

systemctl start nfs-server
systemctl enable nfs-server

kubernetes集群所有节点安装nfs客户端

yum -y install nfs-utils

使用helm部署NFS Subdir External Provisioner

为了操作方便，我直接在master上操作。

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

helm repo update

helm search repo nfs-subdir-external-provisioner
NAME                                                    CHART VERSION   APP VERSION     DESCRIPTION
nfs-subdir-external-provisioner/nfs-subdir-exte...      4.0.13          4.0.2           nfs-subdir-external-provisioner is an automatic...

helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --version 4.0.13

tar -xvf nfs-subdir-external-provisioner-4.0.13.tgz

cd nfs-subdir-external-provisioner/

修改values.yaml

...
replicaCount: 2 # 修改副本数

image:
  repository: willdockerhub/nfs-subdir-external-provisioner # 修改镜像仓库
  tag: v4.0.2

nfs:
  server: 192.168.238.145  # nfs服务器地址
  path: /nfs-data  # 路径

storageClass:
  defaultClass: true  # 开启默认存储类
...

开始安装,将其安装至kube-system命名空间下

helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner -f /root/nfs-subdir-external-provisioner/values.yaml

查看安装

kubectl get pods -n kube-system
NAME                                               READY   STATUS    RESTARTS   AGE
...
nfs-subdir-external-provisioner-677789d968-lsvp5   1/1     Running   0          21s
nfs-subdir-external-provisioner-677789d968-xvgs8   1/1     Running   0          21s

查看是否创建了默认存储类

kubectl get sc
NAME                   PROVISIONER                                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-subdir-external-provisioner   Delete          Immediate           true                   64s

测试

自定义存储类

创建yaml文件

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: cluster.local/nfs-subdir-external-provisioner  # 可以是其他名字,但是必须是deployment环境变量 PROVISIONER_NAME 的值
parameters:
  archiveOnDelete: "true"  # false表示不存档，即删除数据，true表示存档，即重命名路径
allowVolumeExpansion: true  # 是否允许pvc扩容

查看创建的storageClass

kubectl get sc
NAME                   PROVISIONER                                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage    cluster.local/nfs-subdir-external-provisioner   Delete          Immediate           true                   4s
nfs-client (default)   cluster.local/nfs-subdir-external-provisioner   Delete          Immediate           true                   39m

创建一个tomcat应用进行测试

helm repo add bitnami https://charts.bitnami.com/bitnami

helm repo update

helm install tomcat bitnami/tomcat \
  --namespace=test01 \
  --create-namespace \
  --set global.storageClass=managed-nfs-storage \  # 此处使用自定义的storageClass，也可用使用默认的
  --set replicaCount=2 \
  --set service.type=NodePort \
  --set service.nodePort=30089

1. 查看是否自动创建pv

kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM           STORAGECLASS          REASON   AGE
pvc-6300d49b-f3f1-4874-ad77-f174c0acf48f   8Gi        RWO            Delete           Bound    test01/tomcat   managed-nfs-storage            27s

2. 查看应用是否创建pvc

kubectl get pvc -n test01
NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
tomcat   Bound    pvc-6300d49b-f3f1-4874-ad77-f174c0acf48f   8Gi        RWO            managed-nfs-storage   40s

3. 查看pod状态

kubectl get pods -n test01 -o wide
NAME                      READY   STATUS    RESTARTS   AGE     IP           NODE      NOMINATED NODE   READINESS GATES
tomcat-697685bcd9-mzdqn   1/1     Running   0          4m37s   10.244.1.6   k8s-n01   <none>           <none>
tomcat-697685bcd9-z9cmt   1/1     Running   0          4m37s   10.244.2.7   k8s-n02   <none>           <none>

4. 查看service

kubectl get svc -n test01
NAME     TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
tomcat   NodePort   10.1.161.183   <none>        80:30089/TCP   5m33s

5. 访问tomcat

http://192.168.238.147:30089/

如下图：

6. 到nfs服务器上验证文件是否存储归档

tree /nfs-data/ -L 3
/nfs-data/
├── archived-test01-tomcat-pvc-f9bca252-1150-4cd9-bb9b-c29f8c6b2042
└── test01-tomcat-pvc-6300d49b-f3f1-4874-ad77-f174c0acf48f
    └── webapps
        ├── docs
        ├── examples
        ├── host-manager
        ├── manager
        └── ROOT

8 directories, 0 files

参考文档：
[1]: https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner