基于 PVC实现 NFS 容器化

现在的应用服务基本都会使用到共享配置、共享文件、共享日志等服务，NFS （Network File System）作为当前互联网系统架构中最常用的数据存储服务，非常适合于分布式系统中，本文主要简介 NFS 容器化搭建以及PVC 的划分。

创建 StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: auto-high-cbs
parameters:
  paymode: PREPAID
  type: CLOUD_PREMIUM
  zone: "150001"
provisioner: cloud.tencent.com/qcloud-cbs
reclaimPolicy: Retain
volumeBindingMode: Immediate

• 由于设计多个服务或POD之间共享 NFS 服务，需要硬盘读写具有更高的性能. 本次采用的是预付费(PREPAID)模式下的高性能云硬盘(CLOUD_PREMIUM), 此外腾讯云还支持普通云硬盘(CLOUD_BASIC)、 SSD (CLOUD_SSD).
• 由于 NFS 一般用于存储配置等信息，则文件策略使用Retain(保留).

创建 PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"d1ef5ddc-ba81-11e9-9b04-1a812535d98f","leaseDurationSeconds":15,"acquireTime":"2019-09-06T01:02:07Z","renewTime":"2019-09-06T01:02:15Z","leaderTransitions":0}'
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: cloud.tencent.com/qcloud-cbs
  finalizers:
  - kubernetes.io/pvc-protection
  name: nfs-pvc
  namespace: common
spec:
  accessModes:
  - ReadWriteOnce
  dataSource: null
  resources:
    requests:
      storage: 200Gi
  storageClassName: auto-high-cbs
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 200Gi
  phase: Bound

创建出200GB 的内存，并设置权限为ReadWriteOnce

构建 NFS 基础镜像

FROM alpine:latest
LABEL maintainer "Tayl0r Dang(dangtuo888@gmail.com)"
LABEL source "https://github.com/dtcka/nfs-server-alpine"
LABEL branch "master"

RUN apk update && apk add tzdata && \
    cp /usr/share/zoneinfo/Etc/GMT+8 /etc/localtime && \
    apk add --no-cache --update --verbose nfs-utils bash iproute2 && \
    rm -rf /var/cache/apk /tmp /sbin/halt /sbin/poweroff /sbin/reboot && \
    mkdir -p /var/lib/nfs/rpc_pipefs /var/lib/nfs/v4recovery && \
    echo "rpc_pipefs    /var/lib/nfs/rpc_pipefs rpc_pipefs      defaults        0       0" >> /etc/fstab && \
    echo "nfsd  /proc/fs/nfsd   nfsd    defaults        0       0" >> /etc/fstab 

COPY exports /etc/
COPY nfsd.sh /usr/bin/nfsd.sh
COPY .bashrc /root/.bashrc

RUN chmod +x /usr/bin/nfsd.sh

ENTRYPOINT ["/usr/bin/nfsd.sh"]

基于 alpine 制作最精简的 nfs-server 镜像，并且添加了当前服务时区 GMT+8

创建 NFS Deployment

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "13"
  generation: 35
  labels:
    k8s-app: nfs-server
    qcloud-app: nfs-server
  name: nfs-server
  namespace: common
spec:
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: nfs-server
      qcloud-app: nfs-server
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: nfs-server
        qcloud-app: nfs-server
    spec:
      containers:
      - env:
        - name: PATH
          value: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        - name: SHARED_DIRECTORY
          value: /nfs-pvc
        image: xxx/nfs-alpine:1.0.0
        imagePullPolicy: Always
        name: nfs
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 250m
            memory: 256Mi
        securityContext:
          privileged: true
          procMount: Default
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /nfs-pvc
          name: data
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: qcloudregistrykey
      - name: tencenthubkey
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: nfs-pvc

• 设置服务启动 POD 资源：request(CPU--MEM): 250m,256Mi, limit(CPU--MEM): 500m,512Mi
• 添加PVC Volumes：nfs-pvc，挂载 PVC volumeMounts
• 指定 NFS 存储路径：SHARED_DIRECTORY :/nfs-pvc (自定义目录)

添加自动伸缩 HorizontalPodAutoscaler

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  labels:
    qcloud-app: hpa-bwsrr9sttds0
  name: hpa-bwsrr9sttds0
  namespace: common
spec:
  maxReplicas: 4
  metrics:
  - pods:
      metricName: k8s_pod_rate_cpu_core_used_limit
      targetAverageValue: "80"
    type: Pods
  - pods:
      metricName: k8s_pod_rate_mem_usage_limit
      targetAverageValue: "80"
    type: Pods
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1beta2
    kind: Deployment
    name: nfs-server

基于两项指标作为容器扩展的要素：

1. CPU占用超过 limit 值80%
2. 内存使用率超过 limit 值80%

开放VPC 下服务入口

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.kubernetes.io/loadbalance-id: lb-8le93js3
    service.kubernetes.io/qcloud-loadbalancer-clusterid: cls-7sxwq4u8
    service.kubernetes.io/qcloud-loadbalancer-internal-subnetid: subnet-oatgu5wo
  name: nfs-server
  namespace: common
spec:
  clusterIP: 172.16.255.212
  externalTrafficPolicy: Cluster
  ports:
  - name: tcp-2049-2049
    nodePort: 30034
    port: 2049
    protocol: TCP
    targetPort: 2049
  - name: tcp-20048-20048
    nodePort: 30659
    port: 20048
    protocol: TCP
    targetPort: 20048
  - name: tcp-111-111
    nodePort: 32241
    port: 111
    protocol: TCP
    targetPort: 111
  selector:
    k8s-app: nfs-server
    qcloud-app: nfs-server
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 172.26.0.x

nfs会开启3个服务:

• portmapper : 做端口映射的 ( 默认使用 111 端口 )
• mountd : 管理NFS的文件系统 ( 默认使用 20048 端口 可在 /etc/services 中查看到 )
• nfs : 管理客户端登录 ( 默认使用 2049 端口 )

---

至此，NFS 服务已搭建完成

附：

• 基于腾讯云 TKE2服务搭建，不同云厂商或原生 K8S 会有所不同
• PVC访问模式：
  • ReadWriteOnce —— 该volume只能被单个节点以读写的方式映射
  • ReadOnlyMany —— 该volume可以被多个节点以只读方式映射
  • ReadWriteMany —— 该volume只能被多个节点以读写的方式映射
• PVC 回收策略:
  • Retain：手动回收
  • Recycle：需要擦出后才能再使用
  • Delete：相关联的存储资产，如AWS EBS，GCE PD，Azure Disk，or OpenStack Cinder卷都会被删除

    更多内容请访问【云原生建筑师】https://blog.dtcka.com