k8s部署redis集群（方案1）

目录

一、创建configmap

二、创建redis集群

三、集群初始化

四、验证集群省略、补充

1.如果整个 redis 集群的 pod 全部都挂掉了，pod自动拉起后，集群不可用，需要重建集群。

2. 重建集群的方法1: 删除 redis 集群所有的资源，然后重新创建 redis 集群

3.重建集群的方法2: 在原有 redis 集群的基础上进行修复

---

前置条件：创建pv，我这里使用的是nfs，做sc

配置文件：

[root@master1 k8s-public-redis]# tree
.
├── redis-cluster-configmap.yml
└── redis-cluster-statefulset.yml

一、创建configmap

redis 配置文件使用 configmap 方式进行挂载， fix-ip.sh 脚本的作用用于当 redis 集群某 pod 重建后 Pod IP 发生变化，在 /data/nodes.conf 中将新的 Pod IP 替换原 Pod IP

vim redis-cluster-configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
  namespace: lishanbin-service
data:
  fix-ip.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${POD_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e '/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/'${POD_IP}'/' ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |
    bind 0.0.0.0
    protected-mode yes
    port 6379
    tcp-backlog 2048
    timeout 0
    tcp-keepalive 300
    daemonize no
    supervised no
    pidfile /var/run/redis.pid
    loglevel notice
    logfile /data/redis.log
    databases 16
    always-show-logo yes
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /data
    masterauth xxxx
    replica-serve-stale-data yes
    replica-read-only no
    repl-diskless-sync no
    repl-diskless-sync-delay 5
    repl-disable-tcp-nodelay no
    replica-priority 100
    requirepass 985211@LiShanBin
    maxclients 32768
    #maxmemory 6g
    maxmemory-policy allkeys-lru
    lazyfree-lazy-eviction no
    lazyfree-lazy-expire no
    lazyfree-lazy-server-del no
    replica-lazy-flush no
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    aof-load-truncated yes
    aof-use-rdb-preamble yes
    lua-time-limit 5000
    cluster-enabled yes
    cluster-config-file /data/nodes.conf
    cluster-node-timeout 15000
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    latency-monitor-threshold 0
    notify-keyspace-events ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size -2
    list-compress-depth 0
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    hll-sparse-max-bytes 3000
    stream-node-max-bytes 4096
    stream-node-max-entries 100
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit replica 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    dynamic-hz yes
    aof-rewrite-incremental-fsync yes
    rdb-save-incremental-fsync yes

二、创建redis集群

vim redis-cluster-statefulset.yml

---
apiVersion: v1
kind: Service
metadata:
  namespace: lishanbin-service
  name: redis-cluster
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: client
  - port: 16379
    targetPort: 16379
    name: gossip
  selector:
    app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: lishanbin-service
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      terminationGracePeriodSeconds: 20
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - redis-cluster
              topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: registry.cn-shenzhen.aliyuncs.com/lishanbin/redis:6.2.6
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        command: ["/etc/redis/fix-ip.sh", "redis-server", "/etc/redis/redis.conf"]
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis/
          readOnly: false
        - name: data
          mountPath: /data
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-cluster
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      storageClassName: "managed-nfs-storage"
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi

tips：使用headless创建的pod，解析地址可以使用nslookup查看

kubectl run -it ubuntu --image=registry.cn-shenzhen.aliyuncs.com/lishanbin/redis-tools-ubuntu:v0.5.1 --restart=Never /bin/bash
 

[root@master1 k8s-public-redis]# kubectl exec -it ubuntu    bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@ubuntu:/# nslookup redis-cluster.lishanbin-service.svc.cluster.local
Server:         10.204.0.10
Address:        10.204.0.10#53

Name:   redis-cluster.lishanbin-service.svc.cluster.local
Address: 10.203.216.139
Name:   redis-cluster.lishanbin-service.svc.cluster.local
Address: 10.203.191.27
Name:   redis-cluster.lishanbin-service.svc.cluster.local
Address: 10.203.150.126
Name:   redis-cluster.lishanbin-service.svc.cluster.local
Address: 10.203.36.194
Name:   redis-cluster.lishanbin-service.svc.cluster.local
Address: 10.203.216.138
Name:   redis-cluster.lishanbin-service.svc.cluster.local
Address: 10.203.36.200

三、集群初始化

1.查看创建的pod/svc等

[root@master1 k8s-public-redis]# kubectl get all -n lishanbin-service
NAME                  READY   STATUS    RESTARTS   AGE
pod/redis-cluster-0   1/1     Running   0          30m
pod/redis-cluster-1   1/1     Running   0          13m
pod/redis-cluster-2   1/1     Running   0          13m
pod/redis-cluster-3   1/1     Running   0          13m
pod/redis-cluster-4   1/1     Running   0          13m
pod/redis-cluster-5   1/1     Running   0          14m

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)              AGE
service/redis-cluster   ClusterIP   None         <none>        6379/TCP,16379/TCP   47m

NAME                             READY   AGE
statefulset.apps/redis-cluster   6/6     47m

2.redis集群初始化

redis集群初始化必须使用ip，先获取ip，上方的nslookup也可批量获取ip。

kubectl get pods -l app=redis-cluster -n lishanbin-service -o jsonpath='{range.items[*]}{.status.podIP}:6379 '

初始化：

kubectl exec -it pod/redis-cluster-0 -n lishanbin-service  -- bash

redis-cli -a xxx --cluster create \
10.203.36.193:6379 \
10.203.36.243:6379 \
10.203.36.244:6379 \
10.203.216.135:6379 \
10.203.36.252:6379 \
10.203.36.250:6379 \
--cluster-replicas 1

或者

redis-cli --cluster create --cluster-replicas 1  10.203.36.253:6379 10.203.36.255:6379 10.203.36.254:6379 10.203.191.22:6379 10.203.216.136:6379 10.203.36.193:6379 -a 'xxxx'

出现下图表示创建成功：

四、验证集群、补充

简单验证集群：

 kubectl exec -it pod/redis-cluster-0 -n lishanbin-service -- bash

redis-cli -h redis-cluster-1.redis-cluster.lishanbin-service.svc.cluster.local -c -a 'xxxx

 

1.如果整个 redis 集群的 pod 全部都挂掉了，pod自动拉起后，集群不可用，需要重建集群。

2. 重建集群的方法1: 删除 redis 集群所有的资源，然后重新创建 redis 集群

(1) 删除 redis 集群中所有的 pod
[root@k8s-master1 redis-cluster]# kubectl delete -f redis-cluster.yml
​
(2) 删除 redis 集群中所有的 pvc(pv)
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-0 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-1 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-2 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-3 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-4 -n redis-cluster
[root@k8s-master1 redis-cluster]# kubectl delete pvc/data-redis-cluster-5 -n redis-cluster
​
(3) 删除 redis 集群中 pod 对应的 nfs 持久化存储目录
[root@k8s_nfs ~]# rm -rf /nfs/archived-redis-cluster-data-redis-cluster-*/
​
(4) 重新创建 redis 集群
[root@k8s-master1 redis-cluster]# kubectl apply -f redis-cluster.yml

3.重建集群的方法2: 在原有 redis 集群的基础上进行修复

(1) 删除 redis 集群中所有的 pod
[root@k8s-master1 redis-cluster]# kubectl delete -f redis-cluster.yml
​
(2) 找到 redis 集群中 pod 对应的 nfs 持久化存储目录后删除 nodes.conf
[root@k8s_nfs ~]# rm -f /nfs/redis-cluster-data-redis-cluster-0*/nodes.conf*
[root@k8s_nfs ~]# rm -f /nfs/redis-cluster-data-redis-cluster-1*/nodes.conf*
[root@k8s_nfs ~]# rm -f /nfs/redis-cluster-data-redis-cluster-2*/nodes.conf*
[root@k8s_nfs ~]# rm -f /nfs/redis-cluster-data-redis-cluster-3*/nodes.conf*
[root@k8s_nfs ~]# rm -f /nfs/redis-cluster-data-redis-cluster-4*/nodes.conf*
[root@k8s_nfs ~]# rm -f /nfs/redis-cluster-data-redis-cluster-5*/nodes.conf*
​
(3) 重新创建 redis 集群
[root@k8s-master1 redis-cluster]# kubectl apply -f redis-cluster.yml