K8S搭建Redis集群

环境准备：

安装好K8S，没搭建K8S的可查看：kubeadm方式搭建K8S集群、RKE搭建K8S【强烈推荐】

需要知道自己的内网IP，为了后面NFS和PV的配置

#通过命令查看内网IP，我的网段为172.31.0.0/24
kubectl get nodes -o wide
#================================================================================================
NAME           STATUS   ROLES                      AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   
172.31.0.110   Ready    controlplane,etcd,worker   7d22h   v1.22.9   172.31.0.110   <none>         
172.31.0.115   Ready    worker                     7d22h   v1.22.9   172.31.0.115   <none>         
172.31.0.82    Ready    worker                     7d22h   v1.22.9   172.31.0.82    <none>        
#================================================================================================

1.创建NFS存储

创建NFS存储主要是为了给Redis提供稳定的后端存储，当Redis的Pod重启或迁移后，依然能获得原先的数据。这里，我们先要创建NFS，然后通过使用PV为Redis挂载一个远程的NFS路径。

yum -y install nfs-utils rpcbind（nfs-utils主包提供文件系统、rpcbind提供rpc协议）

新增/etc/exports文件，用于设置需要共享的路径：

#编辑/etc/exports文件，添加内容
vi /etc/exports
#================================================================================================
/usr/local/kubernetes/redis/pv1 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv2 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv3 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv4 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv5 172.31.0.0/24(rw,sync,no_root_squash)
/usr/local/kubernetes/redis/pv6 172.31.0.0/24(rw,sync,no_root_squash)
#================================================================================================

创建对应文件夹并授权

mkdir -p /usr/local/kubernetes/redis/pv{1..6}
chmod 777 /usr/local/kubernetes/redis/pv{1..6}

启动服务

#启动nfs和rpcbind服务
systemctl enable nfs 
systemctl enable rpcbind
systemctl start nfs
systemctl start rpcbind

exportfs -v
#================================================================================================
/usr/local/kubernetes/redis/pv1
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv2
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv3
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv4
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv5
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/usr/local/kubernetes/redis/pv6
		172.31.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
#================================================================================================

所有节点安装nfs客户端

#在所有节点进行安装操作
yum -y install nfs-utils 

#验证显示被客户端挂载的目录名
showmount -e 172.31.0.110

2.创建PV【server为master节点的内网IP】

vi pv.yaml 
#================================================================================================
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv1
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv1"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-vp2
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv2"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv3
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv3"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv4
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv4"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv5
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv5"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv6
spec:
  storageClassName: redis
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 172.31.0.110
    path: "/usr/local/kubernetes/redis/pv6"
#================================================================================================
#执行创建pv
kubectl create -f pv.yaml 

kubectl get pv

3.创建Configmap

#创建redis文件夹并切换到文件夹下
mkdir /etc/redis && cd /etc/redis

#创建redis.conf文件
vi redis.conf
#================================================================================================
protected-mode no
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
#================================================================================================

#将reids.conf转为configMap文件
kubectl create configmap redis-conf --from-file=redis.conf

#查看创建的configmap:
kubectl describe cm redis-conf
#================================================================================================
Name:         redis-conf
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
redis.conf:
----
protected-mode noprotected-mode no
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379

Events:  <none>
#================================================================================================

4.创建headless service

Headless service是StatefulSet实现稳定网络标识的基础

#创建headless-service.yaml 文件
vi headless-service.yaml 
#================================================================================================
apiVersion: v1
kind: Service
metadata:
  name: redis-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    port: 6379
  clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster
#================================================================================================
#创建
kubectl create -f headless-service.yaml
#查看
kubectl get svc
#可以看到，服务名称为redis-service，其CLUSTER-IP为None，表示这是一个“无头”服务
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes      ClusterIP   10.43.0.1    <none>        443/TCP    7d19h
redis-service   ClusterIP   None         <none>        6379/TCP   5s

5.创建Redis StatefulSet

#创建redis.yaml
vi redis.yaml 
#================================================================================================
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-app
spec:
  serviceName: "redis-service"
  replicas: 6 
  selector: 
    matchLabels:
      app: redis
      appCluster: redis-cluster
  template:
    metadata:
      labels:
        app: redis
        appCluster: redis-cluster
    spec:
      containers:
      - name: redis
        image: "redis:6.0.6"
        command: ["/bin/bash", "-ce", "tail -f /dev/null"]
        command: ["redis-server"]
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"
        ports:
            - name: redis
              containerPort: 6379
              protocol: "TCP"
            - name: cluster
              containerPort: 16379
              protocol: "TCP"
        volumeMounts:
          - name: "redis-conf"
            mountPath: "/etc/redis"
          - name: "redis-data"
            mountPath: "/var/lib/redis"
      volumes:
      - name: "redis-conf"
        configMap:
          name: "redis-conf"
          items:
            - key: "redis.conf"
              path: "redis.conf"
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteMany" ]
      storageClassName: "redis"
      resources:
        requests:
          storage: 1Gi

#================================================================================================
#创建新资源
kubectl create -f redis.yaml
#查看redis pods
kubectl get pods -o wide 
#================================================================================================
NAME          READY   STATUS    RESTARTS   AGE    IP           NODE           NOMINATED NODE   READINESS 
redis-app-0   1/1     Running   0          4m57s   10.42.1.31   172.31.0.115   <none>           <none>
redis-app-1   1/1     Running   0          4m56s   10.42.2.23   172.31.0.82    <none>           <none>
redis-app-2   1/1     Running   0          4m54s   10.42.0.21   172.31.0.110   <none>           <none>
redis-app-3   1/1     Running   0          4m51s   10.42.1.32   172.31.0.115   <none>           <none>
redis-app-4   1/1     Running   0          4m50s   10.42.2.24   172.31.0.82    <none>           <none>
redis-app-5   1/1     Running   0          4m48s   10.42.0.22   172.31.0.110   <none>           <none>
#================================================================================================
#查看PV
kubectl get pv
#================================================================================================
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS 
nfs-pv1   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-2   redis     
nfs-pv3   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-0   redis     
nfs-pv4   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-4   redis     
nfs-pv5   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-5   redis     
nfs-pv6   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-1   redis     
nfs-vp2   2Gi        RWX            Retain           Bound    default/redis-data-redis-app-3   redis    
#================================================================================================

6.初始化Redis集群

初始化集群采用redis-trib这个工具，直接yum安装redis-trib这个软件，执行初始化

yum -y install redis-trib

redis-trib create --replicas 1 10.42.1.31:6379 10.42.2.23:6379 10.42.0.21:6379 10.42.1.32:6379 10.42.2.24:6379 10.42.0.22:6379

或docker 容器构建

docker pull inem0o/redis-trib

docker run --rm -ti inem0o/redis-trib create --replicas 1 10.42.1.31:6379 10.42.2.23:6379 10.42.0.21:6379 10.42.1.32:6379 10.42.2.24:6379 10.42.0.22:6379

构建成功

#===============================================进入容器=========================================
kubectl  exec  -it  redis-app-0  -- redis-cli  -c
#===============================================查看集群=========================================
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:82
cluster_stats_messages_pong_sent:80
cluster_stats_messages_sent:162
cluster_stats_messages_ping_received:75
cluster_stats_messages_pong_received:82
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:162
#===============================================查看角色=========================================
127.0.0.1:6379> role
1) "master"
2) (integer) 84
3) 1) 1) "10.42.1.36"
      2) "6379"
      3) "84"
#===============================================查看节点=========================================
127.0.0.1:6379> cluster nodes
fc63373625ab65f480c5343ffca4f7d650af8ee6 10.42.0.25:6379@16379 master - 0 1653709741112 3 connected 10923-16383
c444518310f7be61ace5895605f24fe23fd4625b 10.42.2.28:6379@16379 slave 635a262b8e0b2e340acf68262a73870099f40be6 0 1653709740000 2 connected
e57fff1006a56eea193179f0ffcf91ee91e8b800 10.42.1.36:6379@16379 slave 1fe5682e876057cc1e2346b4434008abb7ed2ab8 0 1653709740512 1 connected
635a262b8e0b2e340acf68262a73870099f40be6 10.42.2.27:6379@16379 master - 0 1653709741000 2 connected 5461-10922
289c8aae1b7093b3ae58e7be9bc6bc75abd8dcef 10.42.0.26:6379@16379 slave fc63373625ab65f480c5343ffca4f7d650af8ee6 0 1653709740612 3 connected
1fe5682e876057cc1e2346b4434008abb7ed2ab8 10.42.1.35:6379@16379 myself,master - 0 1653709739000 1 connected 0-5460

7、创建用于访问Service

vi redis-access-service.yaml 

apiVersion: v1
kind: Service
metadata:
  name: redis-access-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    protocol: "TCP"
    port: 6379
    targetPort: 6379
    nodePort: 6379
  selector:
    app: redis
    appCluster: redis-cluster
  type: NodePort
    

kubectl get svc redis-access-service -o wide

注意：这里我采用的是NodePort直接开放外部端口使用，想采用默认的ClusterId，然后通过Ingress开放端口进行访问也行。