K8S部署redis三主三从标准集群

docker pull redis:6.0

参考文章：

k8s-1.2.3部署redis-cluster+predixy代理集群 - 知乎

1、Redis部署在K8S中注意事项

1.1、Redis是一个有状态应用,不应使用deployment方式部署

        当我们把redis以pod的形式部署在k8s中时，每个pod里缓存的数据都是不一样的，而且pod的IP是会随时变化，这时候如果使用普通的deployment和service来部署redis-cluster就会出现很多问题，因此需要改用StatefulSet + Headless Service来解决。

1.1.1 StatefulSet是k8s中专门用于解决有状态应用部署的一种资源

•    StatefulSet管理的每个Pod都有唯一的文档/网络标识，并且按照数字规律生成，而不是像Deployment中那样名称和IP都是随机的（比如StatefulSet名字为redis，那么pod名就是redis-0, redis-1 ...）
•   StatefulSet中ReplicaSet的启停顺序是严格受控的，操作第N个pod一定要等前N-1个执行完才可以。
•   StatefulSet中的Pod采用稳定的持久化储存，并且对应的PV不会随着Pod的删除而被销毁。

1.1.2 Headless Service就是没有指定Cluster IP的Service

相应的，在k8s的dns映射里，Headless Service的解析结果不是一个Cluster IP，而是它所关联的所有Pod的IP列表。

1.1.3 StatefulSet必须要配合Headless Service使用

它会在Headless Service提供的DNS映射上再加一层，最终形成精确到每个pod的域名映射，格式如下`$(podname).$(headless service name)`，有了这个映射，就可以在配置集群时使用域名替代IP，实现有状态应用集群的管理。

1.2、Redis数据持久化

redis虽然是基于内存的缓存，但还是需要依赖于磁盘进行数据的持久化，以便服务出现问题重启时可以恢复已经缓存的数据。在集群中，我们需要使用共享文件系统 + PV（持久卷）的方式来让整个集群中的所有pod都可以共享同一份持久化储存。

1.3、Redis集群配置

关于通过storageClass创建PVC自动分配PV，请参考文章：K8S使用持久化卷存储到NFS（NAS盘）_雨花山人的博客-CSDN博客

中的4.2 构建能自动分配PV的storageClass 章节。

1.3.1 部署6个POD

直接执行如下文件，可以在zo-dev空间下创建6个redis pod及无头service，并把redis的文件挂载到远程NFS上。

#### redis-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-dev
  namespace: zo-dev
# 注意下面data中不能有任何注释，否则会导致应用启动不了
data:
  fix-ip.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${POD_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 10000
    protected-mode no
    daemonize no
    pidfile /var/run/redis.pid
    port 6379
    tcp-backlog 511
    bind 0.0.0.0
    timeout 3600
    tcp-keepalive 1
    loglevel verbose
    logfile /data/redis.log
    databases 16
    save 900 1
    save 300 10
    save 60 10000
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /data
    requirepass 123456
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    lua-time-limit 20000
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    #rename-command FLUSHALL  ""
    latency-monitor-threshold 0
    notify-keyspace-events ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-entries 512
    list-max-ziplist-value 64
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    hll-sparse-max-bytes 3000
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    aof-rewrite-incremental-fsync yes
---
###  redis-statefulset.yaml  暴露一个无头的集群服务，即不分配集群IP
apiVersion: v1
kind: Service
metadata:
  namespace: zo-dev
  name: redis-cluster-headless
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip  # 集群三主三从互相之间的通信端口
  selector:
    app: redis-cluster-dev
---
# 该部署将集群中的每个节点通过NodePort的方式暴露出去,供终端直接连接以IP+PORT的方式连接
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-node-port
  namespace: zo-dev
spec:
  type: NodePort # 允许通过NODE+IP的方式直接访问
  ports:
    - name: redis
      protocol: TCP
      port: 6379 # 应用占用的端口
      targetPort: 6379 # POD容器端口
      nodePort: 6379 # POD容器占用的node节点端口
  selector:
    app: redis-cluster-dev
status:
  loadBalancer: {}
---
###  redis-statefulset.yaml  部署StatefulSet类型的应用6个POD,把redis的data通过storageclass产生pvc/pv备份到远端的NFS服务器
###  关于storeageclass部署参看 https://blog.csdn.net/tzszhzx/article/details/130241866 的4.2 构建能自动分配PV的storageClass
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: zo-dev
  name: redis-cluster-dev
spec:
  serviceName: redis-cluster-dev
  podManagementPolicy: OrderedReady
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster-dev
  template:
    metadata:
      labels:
        app: redis-cluster-dev
    spec:
      affinity:
        podAntiAffinity: #定义pod反亲和性,目的让6个pod不在同一个主机上,实现均衡分布
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - redis-cluster-dev
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: redis
        image: redis:6.0
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/etc/redis/fix-ip.sh", "redis-server", "/etc/redis/redis.conf"]  # 系统启动时把pod ip写入到redis.conf配置文件中
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis/
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster-dev
          defaultMode: 0755   # 类似chmod 0755
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: "dev-nfs-storage"  # 通过storage-class把data数据备份到NFS上，会自动产生10G PVC声明 / 绑定自动产生的 PV,关于storeageclass部署参看 https://blog.csdn.net/tzszhzx/article/details/130241866 的4.2 构建能自动分配PV的storageClass
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi

1.3.2 设置三主三从

1.3.2.1 配置集群使之能在K8S集群内部访问

k8s-1.2.3部署redis-cluster+predixy代理集群 - 知乎   这篇文章通过自己构建一个redis镜像后通过redis-trib.rb来更便捷的实现redis集群的搭建。

我这里使用是直接进入某个POD，手动执行命令来搭建redis集群。不过可以看到nacos的集群搭建比上面2个方法更简单。K8S实战_雨花山人的博客-CSDN博客 9、K8S部署nacos集群 更加便捷。

有2种方法部署，进入某个POD执行构建集群，

•  查看所有的redis node节点IP（k8s分配的IP）

进入某个容器执行

查看6个POD部署在哪些node上，并可以查看某个pord被分配的IP。

kubectl get pods -n zo-dev  -o wide|grep redis-cluster-dev

更推荐下面的命令一次性可以看到所有的节点部署在哪个服务器上，注意6379后面带个空格

kubectl get pods -l app=redis-cluster-dev -n zo-dev -o jsonpath='{range.items[*]}{.status.podIP}:6379 '

会输出:

10.244.2.13:6379 10.244.16.12:6379 10.244.11.5:6379 10.244.8.25:6379 10.244.17.22:6379 10.244.9.14:6379

• 随便选择一台node节点进入pod，或者通过k8s的dashborad中进入页

kubectl exec -it redis-cluster-dev-4 -n zo-dev -- sh

• 拷贝如上IP端口执行该命令产生一个redis集群，并分配16384个hash槽。

        无密码不需要-a 123456

redis-cli -a 123456 --cluster create --cluster-replicas 1 10.244.8.26:6379 10.244.2.14:6379 10.244.16.13:6379 10.244.17.23:6379 10.244.11.6:6379 10.244.20.3:6379

• 部署成功后，也可以通过该命令查看集群信息

kubectl exec -it redis-cluster-dev-0 -n zo-dev -- redis-cli -a 123456 cluster info

1.4 如上配置可以使得K8S集群内应用访问到redis集群

但如何在K8S集群外可以访问

首先查看所有redis所在的真实PODS

kubectl get pods -l app=redis-cluster-dev -n zo-dev -o wide

逐个机器声明每个redis真实的ip及端口。

kubectl get pods -l app=redis-cluster-dev -n zo-dev -o wide


如下命令需要在每个redis节点执行一遍，其中192.168.10.200为k8s集群master节点的ip
# cluster-announce-ip 对外发布自己的节点ip
kubectl exec -it redis-cluster-dev-0 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-ip 192.168.10.140
# cluster-announce-port 对外发布自己的客户端服务端口
kubectl exec -it redis-cluster-dev-0 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-port 31000
# cluster-announce-bus-port 对外发布自己的集群消息总线端口
kubectl exec -it redis-cluster-dev-0 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-bus-port 32000
kubectl exec -it redis-cluster-dev-0 -n zo-dev -- redis-cli -a 123456 config rewrite


kubectl exec -it redis-cluster-dev-1 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-ip 192.168.10.163
kubectl exec -it redis-cluster-dev-1 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-port 31001
kubectl exec -it redis-cluster-dev-1 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-bus-port 32001
kubectl exec -it redis-cluster-dev-1 -n zo-dev -- redis-cli -a 123456 config rewrite


kubectl exec -it redis-cluster-dev-2 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-ip 192.168.10.164
kubectl exec -it redis-cluster-dev-2 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-port 31002
kubectl exec -it redis-cluster-dev-2 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-bus-port 32002
kubectl exec -it redis-cluster-dev-2 -n zo-dev -- redis-cli -a 123456 config rewrite




kubectl exec -it redis-cluster-dev-3 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-ip 192.168.10.144
kubectl exec -it redis-cluster-dev-3 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-port 31003
kubectl exec -it redis-cluster-dev-3 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-bus-port 32003
kubectl exec -it redis-cluster-dev-3 -n zo-dev -- redis-cli -a 123456 config rewrite


kubectl exec -it redis-cluster-dev-4 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-ip 192.168.10.145
kubectl exec -it redis-cluster-dev-4 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-port 31004
kubectl exec -it redis-cluster-dev-4 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-bus-port 32004
kubectl exec -it redis-cluster-dev-4 -n zo-dev -- redis-cli -a 123456 config rewrite



kubectl exec -it redis-cluster-dev-5 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-ip 192.168.10.166
kubectl exec -it redis-cluster-dev-5 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-port 31005
kubectl exec -it redis-cluster-dev-5 -n zo-dev -- redis-cli -a 123456 config set cluster-announce-bus-port 32005
kubectl exec -it redis-cluster-dev-5 -n zo-dev -- redis-cli -a 123456 config rewrite

1.5 JAVA终端使用redis集群

有2种方式，1、把 redis集群中每个阶段再暴露一个NodePord方式的service

这种比较容易，直接将每个节点暴露出去即可，上面已采用这种方式。

# 该部署将集群中的每个节点通过NodePort的方式暴露出去
apiVersion: v1
kind: Service
metadata:
  name: redis-cluster-service
  namespace: zo-dev
spec:
  type: NodePort # 允许通过NODE+IP的方式直接访问
  ports:
    - name: redis
      protocol: TCP
      port: 6379 # 应用占用的端口
      targetPort: 6379 # POD容器端口
      nodePort: 6379 # POD容器占用的node节点端口
  selector:
    app: redis-cluster-dev
status:
  loadBalancer: {}

此时就可以直接把6个节点的宿主机ip+port暴露给客户端使用了。

可以在任何一台非K8S集群环境中的机器上，只要存在redis，就可以远程连接该集群。

示例：

10.167机器是其中一个redis所在的node，-c 代表使用集群方式连接。

redis-cli -h 192.168.10.167 -p 6379 -c

对于设置了密码的需要执行

192.168.10.167:6379> auth 123456

然后执行

cluster info   及   cluster nodes 2个命令就可以看到集群信息了

192.168.10.167:6379> cluster info
"cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:642
cluster_stats_messages_pong_sent:650
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:1293
cluster_stats_messages_ping_received:650
cluster_stats_messages_pong_received:643
cluster_stats_messages_received:1293

至此一个redis集群就OK了。但在要求非常的时候，这种直接暴露所有node的方式还不是最佳选择。

1.5、Redis-Predixy-cluster配置

1.5.1 为什么要在redis集群前面加个predixy代理？

如上图所视，在redis集群之上还存在一个predixy集群。在predixy集群之上可以加个nginx，负载最终端的请求，对于客户端来说，看起来只是连接了一个单机版的redis，实际上连接的是一个集群。这样做的好处有哪些？

• Redis pod重启可导致IP变化
  • POD重新安装后，NODE IP会变化，此时终端无感知
• Redis处理连接负载高
• 集群扩缩容无感知
• 数据安全风险

更多参考：小米 Redis 的 K8S 容器化部署实践_Cluster

1.5.2 构建predixy镜像并部署

• 下载predixy源码

mkdir -p /opt/docker-build/predixy
cd /opt/docker-build/predixy
wget https://codeload.github.com/joyieldInc/predixy/zip/refs/heads/master
unzip predixy-master.zip
mv predixy-master predixy-1.0.5

• 准备Dockerfile 

FROM centos:7

RUN yum install -y epel-release net-tools
RUN yum install -y redis
RUN yum install -y libstdc++-static gcc gcc-c++ make
RUN mkdir /opt/predixy-1.0.5
RUN mkdir /etc/predixy
COPY ./predixy-1.0.5/src/predixy  /usr/local/bin/
COPY ./predixy-1.0.5/conf/*  /etc/predixy/
ENTRYPOINT ["/usr/local/bin/predixy","/etc/predixy/predixy.conf"]

然后就可以使用该镜像

部署predixy-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: predixy-config
  namespace: k8s-redis  #namespace按自已的修改
data:
  predixy.conf: |

    ################################### GENERAL ####################################
    ## Predixy configuration file example
    ## Specify a name for this predixy service
    ## redis command INFO can get this
    Name Predixy-DefaultNS
    ## Specify listen address, support IPV4, IPV6, Unix socket
    ## Examples:
    # Bind 127.0.0.1:7617
    # Bind 0.0.0.0:7617
    # Bind /tmp/predixy
    ## Default is 0.0.0.0:7617
    Bind 0.0.0.0:7617
    ## Worker threads
    WorkerThreads 4
    ## Memory limit, 0 means unlimited
    ## Examples:
    # MaxMemory 100M
    # MaxMemory 1G
    # MaxMemory 0
    ## MaxMemory can change online by CONFIG SET MaxMemory xxx
    ## Default is 0
    # MaxMemory 0
    ## Close the connection after a client is idle for N seconds (0 to disable)
    ## ClientTimeout can change online by CONFIG SET ClientTimeout N
    ## Default is 0 为0时表示禁止该功能，不主动断开客户端连接
    ClientTimeout 0
    ## IO buffer size
    ## Default is 4096
    # BufSize 4096
    ################################### LOG ########################################
    ## Log file path
    ## Unspecify will log to stdout
    ## Default is Unspecified
    Log /data/predixy.log

    ## LogRotate support

    ## 1d rotate log every day
    ## nh rotate log every n hours   1 <= n <= 24
    ## nm rotate log every n minutes 1 <= n <= 1440
    ## nG rotate log evenry nG bytes
    ## nM rotate log evenry nM bytes
    ## time rotate and size rotate can combine eg 1h 2G, means 1h or 2G roate a time
    ## Examples:
    # LogRotate 1d 2G
    # LogRotate 1d
    LogRotate 1d

    ## Default is disable LogRotate


    ## In multi-threads, worker thread log need lock,
    ## AllowMissLog can reduce lock time for improve performance
    ## AllowMissLog can change online by CONFIG SET AllowMissLog true|false
    ## Default is true
    # AllowMissLog false

    ## LogLevelSample, output a log every N
    ## all level sample can change online by CONFIG SET LogXXXSample N
    LogVerbSample 0
    LogDebugSample 0
    LogInfoSample 100
    LogNoticeSample 1
    LogWarnSample 1
    LogErrorSample 1


    ################################### AUTHORITY ##################################
    # Include auth.conf

    ################################### SERVERS ####################################
    #Include cluster.conf
    # Include sentinel.conf
    # Include try.conf
    ###############################################################################
    #这个clusterserverpool也可以放到cluster.conf文件中，看自已的需求
    ClusterServerPool {
      MasterReadPriority 60
      StaticSlaveReadPriority 50
      DynamicSlaveReadPriority 60
      RefreshInterval 1
      ServerTimeout 1
      ServerFailureLimit 10
      ServerRetryTimeout 1
      KeepAlive 120
      Servers {
        #这个很重要，就是redis-server的集群IP地址，一定要要能取到，可以按我上面的curl方法，能取就行
        + redis-cluster.k8s-redis.svc.cluster.local:6379  
        #+ 10.244.5.43:6379
       }
     }
    ################################### DATACENTER #################################
    ## LocalDC specify current machine dc
    # LocalDC bj

    ## see dc.conf
    # Include dc.conf


    ################################### COMMAND ####################################
    ## Custom command define, see command.conf
    #Include command.conf
    ################################### LATENCY ####################################
    ## Latency monitor define, see latency.conf
    #Include latency.conf

部署PV及PVC

 pv-nfs.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-pv
  namespace: k8s-redis
  labels:
    type: nfs
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: nfs
  nfs:
    path: "/data/volumes"
    server: 192.168.30.114 #k8s-nfs matser
    readOnly: false

pvc-nfs.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc
  namespace: k8s-redis
spec:
  accessModes:
  - ReadWriteMany
  resources:
     requests:
       storage: 5Gi
  storageClassName: nfs

部署predixy无头服务及自动水平扩展的deployment，这个predixy前面应挂nginx进行负载

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
  name: predixy
  namespace: k8s-redis
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      #k8s.cn/name: predixy
      app: predixy
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: predixy
    spec:
      containers:
        - command:
            - predixy
            - /etc/predixy/predixy.conf
          #image: haandol/predixy
          #imagePullPolicy: IfNotPresent
          image: registry-op.test.cn/predixy:1.0.5
          name: predixy
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          resources:
            requests:
              cpu: 100m
              memory: 30Mi
            limits:
              cpu: 100m
              memory: 30Mi
          volumeMounts:
            - mountPath: /etc/predixy/
              name: predixy-config-dir
              readOnly: true
            - mountPath: /data/
              name: predixy-data-dir
      imagePullSecrets:
      - name: registry-op.test.cn
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      volumes:
        - configMap:
            defaultMode: 420
            name: predixy-config
          name: predixy-config-dir
        - name: predixy-data-dir
          persistentVolumeClaim:
            claimName: redis-pvc

---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: predixy
  namespace: k8s-redis
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: predixy
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
       name: cpu
       target:
          type: Utilization
          averageUtilization: 50
  - type: Resource
    resource:
       name: memory
       target:
          type: AverageValue
          averageValue: 80Mi

---
apiVersion: v1
kind: Service
metadata:
  name: predixy
  namespace: k8s-redis
spec:
  externalTrafficPolicy: Cluster
  ports:
    - name: predixy-port
      nodePort: 30617
      port: 7617
      protocol: TCP
      targetPort: 7617
  selector:
    k8s.cn/name: predixy
  sessionAffinity: None
  type: NodePort
  selector:
    app: predixy