![8426eddf066484da37541d637e7917b4.png](https://i-blog.csdnimg.cn/blog_migrate/7f2105a7e14e8b54e69e43b494cee924.png)
全世界只有不到1%的人关注我们
你真的很特别
Redis高可用常见的有两种方式:
主从复制(Replication-Sentinel模式)
Redis集群(Redis-Cluster模式)
在网上查了一圈后,发现大部分operator和helm方式都是主从sentinel模式为主,不具参考,本文结合使用Kubernetes StatefulSets和PersistentVolumes来部署redis-cluster。
StatefulSet是Kubernetes提供的管理有状态应用的负载管理控制器API。在Pods管理的基础上,保证Pods的顺序和一致性。与Deployment一样,StatefulSet也是使用容器的Spec来创建Pod,与之不同StatefulSet创建的Pods在生命周期中会保持持久的标记(例如Pod Name)。
StatefulSet适用于具有以下特点的应用:
具有固定的网络标记(主机名)
具有持久化存储
需要按顺序部署和扩展
需要按顺序终止及删除
需要按顺序滚动更新
PersistentVolume(PV)是集群中由管理员配置的一段网络存储。它是集群中的资源,就像节点是集群资源一样。PV是容量插件,如Volumes,但其生命周期独立于使用PV的任何单个pod。此API对象捕获存储实现的详细信息,包括NFS,iSCSI或特定于云提供程序的存储系统。
PersistentVolumeClaim(PVC)是由用户进行存储的请求。它类似于pod。Pod消耗节点资源,PVC消耗PV资源。Pod可以请求特定级别的资源(CPU和内存)。声明可以请求特定的大小和访问模式(例如,可以一次读/写或多次只读)。
准备redis配置文件 redis.conf
#Redis configuration file#daemonize yes 无需此项,否则pod无法正常启动port 6379tcp-backlog 511timeout 300tcp-keepalive 60loglevel noticedatabases 16#snapshotsave 900 1save 60 100000stop-writes-on-bgsave-error yesrdbchecksum yesdbfilename "dump6379.rdb"dir /var/lib/redis#replicationslave-serve-stale-data yesslave-read-only yesrepl-ping-slave-period 10repl-timeout 600repl-disable-tcp-nodelay norepl-backlog-size 128mbrepl-backlog-ttl 3600slave-priority 100min-slaves-to-write 0min-slaves-max-lag 10repl-diskless-sync yesrepl-diskless-sync-delay 3rdbcompression yes#security authmasterauth Redisrequirepass Redis##aof 请根据自己实际需要是否开启appendonly no#appendfilename "appendonly.aof"#appendfsync alwaysappendfsync everysec#appendfsync nono-appendfsync-on-rewrite noauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mb##slow logslowlog-log-slower-than 100000slowlog-max-len 10240cluster-enabled yescluster-config-file "/var/lib/redis/nodes.conf"cluster-node-timeout 15000cluster-migration-barrier 1# Generated by CONFIG REWRITEprotected-mode no
创建配置卷
[root@worker01 redis]# kubectl create ns test[root@worker01 redis]# kubectl create configmap redis-conf --from-file=redis.conf -n testconfigmap/redis-conf created
创建一个无头服务
[root@worker01 redis]# vim headless-service.yamlapiVersion: v1kind: Servicemetadata: name: redis-service labels: app: redisspec: ports: - name: redis-port port: 6379 clusterIP: None selector: app: redis appCluster: redis-cluster
[root@worker01 redis]# kubectl apply -f headless-service.yaml -n testservice/redis-service created
创建statefulset
[root@worker01 test]# vim statefulset.yaml apiVersion: apps/v1kind: StatefulSetmetadata: name: redis-clusterspec: serviceName: "redis-service" selector: matchLabels: app: redis replicas: 6 template: metadata: labels: app: redis appCluster: redis-cluster annotations: ovn.kubernetes.io/ip_pool: 172.18.0.111,172.18.0.112,172.18.0.113,172.18.0.114,172.18.0.115,172.18.0.116 spec: terminationGracePeriodSeconds: 20 containers: - name: redis image: redis:5.0.7 command: - "redis-server" args: - "/etc/redis/redis.conf" resources: requests: cpu: "500m" memory: "1024Mi" limits: cpu: "2000m" memory: "2048Mi" ports: - name: redis containerPort: 6379 protocol: "TCP" - name: cluster containerPort: 16379 protocol: "TCP" volumeMounts: - name: "redis-conf" mountPath: "/etc/redis" - name: "redis-data" mountPath: "/var/lib/redis" volumes: - name: "redis-conf" configMap: name: "redis-conf" items: - key: "redis.conf" path: "redis.conf" volumeClaimTemplates: - metadata: name: redis-data spec: storageClassName: "xsky-rbd" accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 8G [root@worker01 test]# kubectl apply -f statefulset.yaml -n teststatefulset.apps/redis-cluster created
注意我们这里指定了地址池172.18.0.111-116,其实我们是使用了开源网络插件kube-ovn提供的功能,给statefulset提供固定IP;
annotations: ovn.kubernetes.io/ip_pool: 172.18.0.111,172.18.0.112,172.18.0.113,172.18.0.114,172.18.0.115,172.18.0.116
因为StatefulSet有状态服务,对网络标示的固定有更高的要求,Kube-OVN基于此做了特殊的强化:
Pod 会按顺序分配ovn.kubernetes.io/ip_pool 中的 IP。例如 StatefulSet 的名字为 web,则 web-1 会使用 ovn.kubernetes.io/ip_pool 中的第一个 IP, web-2 会使用第二个 IP,以此类推。
StatefulSet Pod 在更新或删除的过程中 OVN 中的 logical_switch_port 不会删除,新生成的 Pod 直接复用旧的 interface 信息。因此 Pod 可以复用 IP/Mac 及其他网络信息,达到和 StatefulSet Volume 类似的状态保留功能
基于2的能力,对于没有 ovn.kubernetes.io/ip_pool 注解的 StatefulSet,Pod 第一次生成时会随机分配 IP/Mac 之后在整个 StatefulSet 的生命周期内,网络信息都会保持固定。
我们总共创建了6个Redis节点(Pod),其中3个用于master,另外3个分别作为master的slave,Redis的配置通过volume将之前生成的redis-conf这个Configmap,挂载到容器的/etc/redis/redis.conf;Redis的数据存储路径使用volumeClaimTemplates声明(也就是PVC),动态创建StatefulSet中Pod所需的PV。
查看redis各节点
[root@worker01 test]# kubectl get pod -n test -l app=redis -ojsonpath='{range.items[*]}{.status.podIP}:6379 '172.18.0.111:6379 172.18.0.112:6379 172.18.0.113:6379 172.18.0.114:6379 172.18.0.115:6379 172.18.0.116:6379
安装redis-trib集群创建工具
[root@worker01 test]# yum install redis-trib -y
redis集群创建
[root@worker01 test]# redis-trib create --replicas 1 172.18.0.111:6379 172.18.0.112:6379 172.18.0.113:6379 172.18.0.114:6379 172.18.0.115:6379 172.18.0.116:6379 >>> Creating cluster[ERR] Sorry, can't connect to node 172.18.0.111:6379
这是因为我们配置文件中加了auth认证,需要修改client.rb文件
[root@worker01 test]# vim /usr/share/gems/gems/redis-3.2.1/lib/redis/client.rb
将nil修改为配置中设置的密码
再次创建集群
[root@worker01 test]# redis-trib create --replicas 1 172.18.0.111:6379 172.18.0.112:6379 172.18.0.113:6379 172.18.0.114:6379 172.18.0.115:6379 172.18.0.116:6379 >>> Creating cluster>>> Performing hash slots allocation on 6 nodes...Using 3 masters:172.18.0.111:6379172.18.0.112:6379172.18.0.113:6379Adding replica 172.18.0.114:6379 to 172.18.0.111:6379Adding replica 172.18.0.115:6379 to 172.18.0.112:6379Adding replica 172.18.0.116:6379 to 172.18.0.113:6379M: e81c7c1162ad55d9c0fe8d54894860138a911e35 172.18.0.111:6379 slots:0-5460 (5461 slots) masterM: 7a1f94dd9fcd9a4a476c5a10d21f26f0a14ffa9a 172.18.0.112:6379 slots:5461-10922 (5462 slots) masterM: 00362faa3262f2630d66bea74c7ddcb1ca691d5d 172.18.0.113:6379 slots:10923-16383 (5461 slots) masterS: 8e43a90b3980f83cb8926e3cad038cc28e24fff9 172.18.0.114:6379 replicates e81c7c1162ad55d9c0fe8d54894860138a911e35S: c6816410c38d5f8a4d895b2fd9d440ab5e68b59c 172.18.0.115:6379 replicates 7a1f94dd9fcd9a4a476c5a10d21f26f0a14ffa9aS: 3d37b251dba1813c8b25959fd4978d4b019c929d 172.18.0.116:6379 replicates 00362faa3262f2630d66bea74c7ddcb1ca691d5dCan I set the above configuration? (type 'yes' to accept): yes>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join....>>> Performing Cluster Check (using node 172.18.0.111:6379)M: e81c7c1162ad55d9c0fe8d54894860138a911e35 172.18.0.111:6379 slots:0-5460 (5461 slots) master 1 additional replica(s)S: c6816410c38d5f8a4d895b2fd9d440ab5e68b59c 172.18.0.115:6379@16379 slots: (0 slots) slave replicates 7a1f94dd9fcd9a4a476c5a10d21f26f0a14ffa9aS: 3d37b251dba1813c8b25959fd4978d4b019c929d 172.18.0.116:6379@16379 slots: (0 slots) slave replicates 00362faa3262f2630d66bea74c7ddcb1ca691d5dM: 00362faa3262f2630d66bea74c7ddcb1ca691d5d 172.18.0.113:6379@16379 slots:10923-16383 (5461 slots) master 1 additional replica(s)M: 7a1f94dd9fcd9a4a476c5a10d21f26f0a14ffa9a 172.18.0.112:6379@16379 slots:5461-10922 (5462 slots) master 1 additional replica(s)S: 8e43a90b3980f83cb8926e3cad038cc28e24fff9 172.18.0.114:6379@16379 slots: (0 slots) slave replicates e81c7c1162ad55d9c0fe8d54894860138a911e35[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.
集群信息查看
创建用于访问Service前面我们创建了用于实现StatefulSet的Headless Service,但该Service没有Cluster Ip,因此不能用于外界访问。所以,我们还需要创建一个Service,专用于为Redis集群提供访问和负载均衡:
[root@worker01 test]# vim redis-access-service.yaml apiVersion: v1kind: Servicemetadata: name: redis-access-service labels: app: redisspec: ports: - name: redis-port protocol: "TCP" port: 6379 targetPort: 6379 selector: app: redis appCluster: redis-cluster
该Service名称为 redis-access-service
,在K8S集群中暴露6379端口,会对labels name
为app: redis
或appCluster: redis-cluster
的pod进行负载均衡。
在K8S集群中,所有应用都可以通过10.64.59.29:6379
来访问Redis集群。
高可用测试
删除pod节点redis-cluster-0(172.18.0.111) ,观察nodes.conf文件变化,可以看到原来的0.111的slave节点0.114变成了master,0.111起来后变成了新的slave
写在最后:
在实际生产环境中,最好再配置pod的反亲和性,将POD分散在不同的主机或者拓扑域中,以提高服务本身的稳定性。
END
木瓜
投我以木瓜,报之以琼琚。匪报也,永以为好也!
投我以木桃,报之以琼瑶。匪报也,永以为好也!
投我以木李,报之以琼玖。匪报也,永以为好也!