redis使用本地磁盘做pvc

1 local 卷介绍

https://kubernetes.io/zh-cn/docs/concepts/storage/volumes/
https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes/#local

local 卷所代表的是某个被挂载的本地存储设备，例如磁盘、分区或者目录。

local 卷只能用作静态创建的持久卷。不支持动态配置。

与 hostPath 卷相比，local 卷能够以持久和可移植的方式使用，而无需手动将 Pod 调度到节点。系统通过查看 PersistentVolume 的节点亲和性配置，就能了解卷的节点约束。

然而，local 卷仍然取决于底层节点的可用性，并不适合所有应用程序。 如果节点变得不健康，那么 local 卷也将变得不可被 Pod 访问。使用它的 Pod 将不能运行。 使用 local 卷的应用程序必须能够容忍这种可用性的降低，以及因底层磁盘的耐用性特征而带来的潜在的数据丢失风险。

2 部署

设置 label

kubectl label node qa-gpu026 redis=true

如果不设置 label ，那么 pvc 会一直处于 Pending 状态，报错如下

0/3 nodes are available: 3 node(s) didn't find available persistent volumes to bind.

redis 部署文件

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: redis-sc
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain


---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: redis-sc
  hostPath:
    path: {{redis_base_path}}/redis
    type: DirectoryOrCreate
  nodeAffinity:    # Local Persistent Volume（本地持久化卷），必须使用 nodeAffinity 来跟某个具体的节点绑定
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: redis
              operator: In
              values:
                - "true"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc
  namespace: infra
spec:
  storageClassName: redis-sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: infra
  labels:
    app: redis
spec:
  type: ClusterIP
  ports:
    - name: redis
      port: 6379
      targetPort: 6379
      #nodePort: 30379
    - name: metrics
      port: 9121
      targetPort: 9121
      #nodePort: 31121
  selector:
    app: redis

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: redis-config
  namespace: infra
  labels:
    app: redis
data:
  redis.conf: |-
    dir /data
    port 6379
    bind 0.0.0.0
    appendonly yes
    protected-mode no
    requirepass {{redis_password}}
    pidfile /data/redis-6379.pid
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: infra
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9121"
      labels:
        app: redis
    spec:
      nodeSelector:
        redis: "true"
      # 进行初始化操作，修改系统配置，解决 Redis 启动时提示的警告信息
      initContainers:
        - name: system-init
          image: {{init_image}}
          imagePullPolicy: IfNotPresent
          command:
            - "sh"
            - "-c"
            - "echo 2000 > /proc/sys/net/core/somaxconn && echo never > /sys/kernel/mm/transparent_hugepage/enabled"
          securityContext:
            privileged: true
            runAsUser: 0
          volumeMounts:
            - name: sys
              mountPath: /sys
              readOnly: false
      containers:
        - name: redis
          image: {{redis_image}}
          command:
            - "sh"
            - "-c"
            - "redis-server /usr/local/etc/redis/redis.conf"
          ports:
            - containerPort: 6379
          resources:
            limits:
              cpu: 4
              memory: 64Gi
            requests:
              cpu: 0.1
              memory: {{memory_request}}
          livenessProbe:
            tcpSocket:
              port: 6379
            initialDelaySeconds: 300
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            tcpSocket:
              port: 6379
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          volumeMounts:
            - name: data
              mountPath: /data
            - name: config
              mountPath:  /usr/local/etc/redis/redis.conf
              subPath: redis.conf
        - name: redis-exporter
          image: {{redis_exporter_image}}
          args: ["-redis.password", "{{redis_password}}"]
          resources:
            requests:
              cpu: '0.1'
              memory: 100Mi
          ports:
            - containerPort: 9121
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: redis-pvc
        - name: config
          configMap:
            name: redis-config
        - name: sys
          hostPath:
            path: /sys

3 结果

sc

root@qa-gpu026:~# kubectl get sc
NAME                   PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
redis-sc               kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  16h
sc-mysql-single-node   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  16h
swfs-master-sc         kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  16h
swfs-volume-sc         kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  16h
zk-local-volume        kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  16h
root@qa-gpu026:~#

pv

root@qa-gpu026:~# kubectl get pv redis-pv
NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
redis-pv   100Gi      RWO            Retain           Bound    infra/redis-pvc   redis-sc                16h
root@qa-gpu026:~#

pvc

root@qa-gpu026:~# kubectl -n infra get pvc  redis-pvc
NAME        STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-pvc   Bound    redis-pv   100Gi      RWO            redis-sc       16h
root@qa-gpu026:~#

4 问答

问：本地磁盘可以做sc（StorageClass）么

答：可以，有 local storage class，不过，这个 local 稍微有点特殊，他是 lazy provision 的（volumeBindingMode: WaitForFirstConsumer），需要创建 pod 引用才可以（waiting for first consumer to be created before binding）处于 Bound 状态

问：和nfs，cfs有啥区别

答：local 和 host path 一样，是直接做本地 mount，其他都是 网络 fs，走 fuse 挂的

注意：Local Persistent Volume（本地持久化卷），必须使用 nodeAffinity 来跟某个具体的节点绑定