k8s部署redis cluster

环境信息：

k8s：

k8s版本：1.23.17

master01：192.168.66.50

master02：192.168.66.55

master03：192.168.66.56

etcd01：192.168.66.58

etcd02：192.168.66.59

etcd03：192.168.66.57

ingress01：192.168.66.52

node01：192.168.66.51

node02：192.168.66.53

存储节点使用nfs存储：

192.168.66.40

redis是有状态的服务，所以我们在k8s部署redis cluster需要statefuset控制器部署，准备部署reids cluster是三主三从的集群

1：部署reids cluster的前期准备：

创建nfs共享存储

 在192.168.66.40服务器上部署nfs

yum  install nfs-utils rpcbind

创建共享存储目录，存储路径：

mkdir -pv /date/redis{0..5}

ls  -lh /data

[root@docker ~]# ls -lh /data
total 0
drwxr-xr-x. 2 root root 59 Jul 24 21:53 redis0
drwxr-xr-x. 2 root root 59 Jul 24 21:54 redis1
drwxr-xr-x. 2 root root 59 Jul 24 21:54 redis2
drwxr-xr-x. 2 root root 59 Jul 24 21:53 redis3
drwxr-xr-x. 2 root root 59 Jul 24 21:54 redis4
drwxr-xr-x. 2 root root 59 Jul 24 21:54 redis5

验证：在k8s上的node节点上测试，是否可以挂在nfs共享盘

# showmount -e 192.168.66.40

 

2：在k8s上面开始部署redis cluster：

a:创建pv，sc，headless service和comfigmap

pv创建，编写yaml文件,需要创建6个pv，每个redis节点挂在一个pv

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-redis-pv0
spec:
  storageClassName: redis-sc
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.66.40
    path: "/data/redis0"

#kubectl apply -f redis-pv.yaml

b:创建sc，即StorageCluster类型存储，之后的pvc就是依赖于StorageCluster创建

[root@master03 redis_cluster_yaml]# vim redis-sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: redis-sc
provisioner: nfs-storage
~                        

c:创建 headless service，headless service是redis依赖访问的方式，必须创建

apiVersion: v1
kind: Service
metadata:
  name: redis-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-service
    port: 6379
  clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster

d:创建configmap，创建的configmap主要是挂在redis配置文件

#vim redis.config

[root@master03 redis_cluster_yaml]# vim redis.conf

appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
bind 0.0.0.0
daemonize no

基于redis.conf创建configmap，作为redis cluster的配置文件

# kubectl create configmap redis-config --from-file=redis.conf

d：查看创建的svc，pv，sc，configmap

[root@master03 redis_cluster_yaml]# kubectl get pv,svc,sc,cm
NAME                             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
persistentvolume/nfs-redis-pv0   2Gi        RWX            Retain           Available           redis-sc                27s
persistentvolume/nfs-redis-pv1   2Gi        RWX            Retain           Available           redis-sc                27s
persistentvolume/nfs-redis-pv2   2Gi        RWX            Retain           Available           redis-sc                27s
persistentvolume/nfs-redis-pv3   2Gi        RWX            Retain           Available           redis-sc                27s
persistentvolume/nfs-redis-pv4   2Gi        RWX            Retain           Available           redis-sc                27s
persistentvolume/nfs-redis-pv5   2Gi        RWX            Retain           Available           redis-sc                27s

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/kubernetes      ClusterIP   10.96.0.1    <none>        443/TCP    33h
service/redis-service   ClusterIP   None         <none>        6379/TCP   41s

NAME                                   PROVISIONER   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/redis-sc   nfs-storage   Delete          Immediate           false                  2m41s

NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      33h
configmap/redis-config       1      103s

 3：创建redis pods，编写yaml文件

            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - redis
              topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: redis:6.2.6
        command:
          - "redis-server"
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"

       resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
            - name: redis
              containerPort: 6379
              protocol: "TCP"
            - name: cluster
              containerPort: 16379
              protocol: "TCP"
        volumeMounts:
          - name: "redis-conf"
            mountPath: "/etc/redis"
          - name: "redis-data"
            mountPath: "/var/lib/redis"
          - name: localtime                                               #挂载本地时间
            mountPath: "/etc/localtime"
      volumes:
      - name: "redis-conf"
        configMap:
          name: "redis-config"
          items:
            - key: "redis.conf"
              path: "redis.conf"
      - name: "localtime"                                         #挂载本地时间
        hostPath:
          path: "/usr/share/zoneinfo/Asia/Shanghai"
          type: File
  volumeClaimTemplates:
  - metadata:

      name: redis-data
    spec:
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 2Gi
      storageClassName: redis-sc

# kubectl apply -f redis-cluster.yaml

#kubectl get pods

在default的名称空间下，创建出6个redis pods

4：在同一名称空间下，再起一个ubuntu pods，来让redis节点加入redis cluster中

# kubectl run -it ubuntu --image=ubuntu --restart=Never /bin/bash

ubuntu的版本是22.04，更换镜像仓库源，安装必须的软件包

root@ubuntu:/# cat << EOF > /etc/apt/sources.list && apt update
deb http://mirrors.ustc.edu.cn/ubuntu/ jammy main restricted universe multiverse
deb-src http://mirrors.ustc.edu.cn/ubuntu/ jammy main restricted universe multiverse
deb http://mirrors.ustc.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse
deb-src http://mirrors.ustc.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse
deb http://mirrors.ustc.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse
deb-src http://mirrors.ustc.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse
deb http://mirrors.ustc.edu.cn/ubuntu/ jammy-security main restricted universe multiverse
deb-src http://mirrors.ustc.edu.cn/ubuntu/ jammy-security main restricted universe multiverse
deb http://mirrors.ustc.edu.cn/ubuntu/ jammy-proposed main restricted universe multiverse
deb-src http://mirrors.ustc.edu.cn/ubuntu/ jammy-proposed main restricted universe multiverse
EOF

安装软件包：

# apt-get install -y libncursesw5 libreadline6 libtinfo5 --allow-remove-essential

# apt-get install -y libpython2.7-stdlib python2.7 python-pip redis-tools dnsutils

使用pip2安装组件reids cluster的工具redis-trib.py

# pip2 install redis-trib==0.5.1

使用nslookup，测试redis pods的是否能被解析

#nslookup -A redis-0.redis-service

5：使用redis-trib.py添加集群节点

/# redis-trib.py create \
  `dig +short redis-0.redis-service.default.svc.cluster.local`:6379 \
  `dig +short redis-1.redis-service.default.svc.cluster.local`:6379 \
  `dig +short redis-2.redis-service.default.svc.cluster.local`:6379

给master节点分别添加slave节点：

#redis-trib.py replicate   --master-addr `dig +short redis-0.redis-service.default.svc.cluster.local`:6379  --slave-addr `dig +short redis-3.redis-service.default.svc.cluster.local`:6379

#redis-trib.py replicate   --master-addr `dig +short redis-1.redis-service.default.svc.cluster.local`:6379  --slave-addr `dig +short redis-4.redis-service.default.svc.cluster.local`:6379

#redis-trib.py replicate   --master-addr `dig +short redis-2.redis-service.default.svc.cluster.local`:6379  --slave-addr `dig +short redis-5.redis-service.default.svc.cluster.local`:6379

 

6：验证redis cluster是否正常

# redis-cli

127.0.0.1:6379> cluster nodes

127.0.0.1:6379> cluster info