目录
一、什么是redis Cluster集群
Redis集群是一种基于数据分片(sharding)的分布式缓存和存储系统。是一个由多个主从节点群组成的分布式服务集群,它具有复制、高可用和分片特性。Redis集群不需要sentinel哨兵也能完成节点移除和故障转移的功能。需要将每个节点设置成集群模式,这种集群模式没有中心节点,可水平扩展,据官方文档称可以线性扩展到上万个节点(官方推荐不超过1000个节点)。redis集群的性能和高可用性均优于之前版本的哨兵模式,且集群配置非常简单。它是Redis为了应对大规模数据存储、高并发访问和高可用性需求而提供的原生集群解决方案。
二、集群架构图
redis 3.0版本之后推出了无中心架构的redis cluster机制,在无中心的redis集群当中,其每个节点保存当前节点数据和整个集群状态,每个节点都和其他所有节点连接
三、redis Cluster部署架构
1、测试环境
3台服务器,每台服务器启动6379和6380两个redis 服务实例
2、生产环境
6台服务器,分别是三组master/slave,适用于生产环境
四、原生命令手动部署Redis Cluster
- 在所有节点安装redis,并配置开启cluster功能
- 各个节点执行meet,实现所有节点的相互通信
- 为各个master 节点指派槽位范围
- 指定各个节点的主从关系
1、环境准备
角色 | 主机名 | IP地址 |
---|---|---|
主节点 | m1 | 192.168.112.10 |
主节点 | m2 | 192.168.112.20 |
主节点 | m3 | 192.168.112.30 |
从节点 | s1 | 192.168.112.40 |
从节点 | s2 | 192.168.112.50 |
从节点 | s3 | 192.168.112.60 |
所有主机都安装了redis
2、为所有节点启用redis集群支持
bash
echo "cluster-enabled yes" >> /apps/redis/etc/redis.conf echo "masterauth centos" >> /apps/redis/etc/redis.conf
bash
[root@m1 ~]# egrep -v "^#|^$" /apps/redis/etc/redis.conf bind 0.0.0.0 protected-mode yes port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize no supervised no pidfile /apps/redis/run/redis_6379.pid loglevel notice logfile /apps/redis/log/redis_6379.log databases 16 always-show-logo yes save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir /apps/redis/data/ replica-serve-stale-data yes replica-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-disable-tcp-nodelay no replica-priority 100 requirepass centos lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes aof-use-rdb-preamble yes lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 stream-node-max-bytes 4096 stream-node-max-entries 100 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 dynamic-hz yes aof-rewrite-incremental-fsync yes rdb-save-incremental-fsync yes cluster-enable yes masterauth centos
bash
#重启redis systemctl restart redis
3、执行meet操作实现互相通信
在任意一节点上和其它所有节点进行meet通信,以m1为例
bash
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.20 6379 OK [root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.30 6379 OK [root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.40 6379 OK [root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.50 6379 OK [root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.60 6379 OK
#可以看到所有节点之间可以相互连接通信,以m1为例
bash
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster nodes a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 myself,master - 0 1713451004000 1 connected 69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 master - 0 1713451005000 5 connected 4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 master - 0 1713451005696 4 connected efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713451006719 0 connected c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 master - 0 1713451003656 2 connected b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713451004000 3 connected
4、为每个master节点指派槽位范围
由于没有分配槽位,无法创建key
bash
[root@m1 ~]# redis-cli -a centos --no-auth-warning set name misaki (error) CLUSTERDOWN Hash slot not served
查看当前状态,以m1为例
bash
[root@m1 ~]# redis-cli -a centos --no-auth-warning cluster info cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:0 cluster_current_epoch:5 cluster_my_epoch:1 cluster_stats_messages_ping_sent:495 cluster_stats_messages_pong_sent:469 cluster_stats_messages_meet_sent:5 cluster_stats_messages_sent:969 cluster_stats_messages_ping_received:469 cluster_stats_messages_pong_received:500 cluster_stats_messages_received:969
为每个master 节点指派槽位范围
bash
[root@m1 ~]# vim addslots.sh #!/bin/bash HOST=$1 PORT=$2 START=$3 END=$4 PASS=centos for slot in `seq ${START} ${END}`;do echo "slot: ${slot}" redis-cli -h ${HOST} -p ${PORT} -a ${PASS} --no-auth-warning cluster addslots ${slot} done * 为三个master分配槽位,共16364/3=5461.33333,平均每个master分配5461个槽位 [root@m1 ~]# bash addslots.sh 192.168.175.10 6379 0 5461
当master1分配完槽位后,可以看到下面信息
bash
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster info cluster_state:fail cluster_slots_assigned:5462 cluster_slots_ok:5462 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:1 cluster_current_epoch:5 cluster_my_epoch:1 cluster_stats_messages_ping_sent:745 cluster_stats_messages_pong_sent:691 cluster_stats_messages_meet_sent:5 cluster_stats_messages_sent:1441 cluster_stats_messages_ping_received:691 cluster_stats_messages_pong_received:750 cluster_stats_messages_received:1441
为剩下的master节点分配槽位
bash
[root@m1 ~]# bash addslots.sh 192.168.112.20 6379 5462 10922 [root@m1 ~]# bash addslots.sh 192.168.112.30 6379 10923 16383
为所有的三个master分配完槽位后,验证信息
bash
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster nodes a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 myself,master - 0 1713452132000 1 connected 0-5461 69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 master - 0 1713452137000 5 connected 4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 master - 0 1713452135000 4 connected efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713452138634 0 connected 10923-16383 c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 master - 0 1713452137000 2 connected 5462-10922 b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713452137612 3 connected
5、指定各个节点的主从关系
通过上面的cluster nodes 查看master的ID信息,执行下面操作,将对应的slave 指定相应的master节点,实现三对主从节点
bash
[root@m1 ~]# redis-cli -h 192.168.112.40 -a centos --no-auth-warning cluster replicate a8f0a6fa7de004d25996eefd652e4bf8fb995386 OK [root@m1 ~]# redis-cli -h 192.168.112.50 -a centos --no-auth-warning cluster replicate c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 OK [root@m1 ~]# redis-cli -h 192.168.112.60 -a centos --no-auth-warning cluster replicate efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a OK
6、查看主节点关系及槽位信息
bash
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster slots 1) 1) (integer) 0 2) (integer) 5461 3) 1) "192.168.112.10" 2) (integer) 6379 3) "a8f0a6fa7de004d25996eefd652e4bf8fb995386" 4) 1) "192.168.112.40" 2) (integer) 6379 3) "b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68" 2) 1) (integer) 10923 2) (integer) 16383 3) 1) "192.168.112.30" 2) (integer) 6379 3) "efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a" 4) 1) "192.168.112.60" 2) (integer) 6379 3) "69f2aee99fc15ddde526dad3cb55b6532c5c6b06" 3) 1) (integer) 5462 2) (integer) 10922 3) 1) "192.168.112.20" 2) (integer) 6379 3) "c3ddc4db15222b0cdb2a64974a87b6bd731d8f84" 4) 1) "192.168.112.50" 2) (integer) 6379 3) "4f5fa3759a45025478aaa062c4290b4b478ab3ce"
7、验证redis Cluster访问
-c 表示以集群方式连接
bash
[root@m1 ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning set name kk OK [root@m1 ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning get name "kk" [root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning get name (error) MOVED 5798 192.168.112.20:6379 [root@m1 ~]# redis-cli -h 192.168.112.20 -a centos --no-auth-warning get name "kk" [root@m1 ~]# redis-cli -h 192.168.112.30 -a centos --no-auth-warning get name (error) MOVED 5798 192.168.112.20:6379
8、测试故障切换
模拟m1主机点宕机
bash
systemctl stop redis
bash
[root@m2 ~]# redis-cli -c -h 192.168.112.20 -a centos --no-auth-warning cluster nodes c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 myself,master - 0 1713489205000 2 connected 5462-10922 69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 slave efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 0 1713489207000 5 connected b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713489207436 6 connected 0-5461 4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 slave c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 0 1713489206000 4 connected a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 master,fail - 1713489156541 1713489154401 1 disconnected efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713489208451 0 connected 10923-16383
可以看到s1已经由slave转变为master,过程大概在5-10s
五、基于官方脚本搭建redis cluster
创建 redis cluster集群的环境准备:
- 每个redis 节点采用相同的硬件配置、相同的密码、相同的redis版本
- 所有redis服务器必须没有任何数据
- 准备6台主机,开启cluster配置
1、哨兵配置
bash
echo "masterauth centos" >> /apps/redis/etc/redis.conf echo "cluster-enabled yes" >> /apps/redis/etc/redis.conf
2、开启集群
bash
systemctl restart redis
# redis-cli --cluster-replicas 1 表示每个master对应一个slave节点
任意一台机器上执行就行
bash
[root@localhost ~]# redis-cli -a centos --no-auth-warning --cluster create \ > 192.168.112.10:6379 192.168.112.20:6379 192.168.112.30:6379 \ > 192.168.112.40:6379 192.168.112.50:6379 192.168.112.60:6379 \ > --cluster-replicas 1 >>> Performing hash slots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 192.168.112.50:6379 to 192.168.112.10:6379 Adding replica 192.168.112.60:6379 to 192.168.112.20:6379 Adding replica 192.168.112.40:6379 to 192.168.112.30:6379 M: c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379 slots:[0-5460] (5461 slots) master M: 43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379 slots:[5461-10922] (5462 slots) master M: e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379 slots:[10923-16383] (5461 slots) master S: 5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379 replicates e621db96b845eb7818eb01a37fccdc7796f8d7de S: bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379 replicates c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 S: 510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379 replicates 43906d49ffe4be7085b4d37f52c50ac7bb4df751 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join .. >>> Performing Cluster Check (using node 192.168.112.10:6379) M: c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379 slots: (0 slots) slave replicates c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 M: e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379 slots:[10923-16383] (5461 slots) master 1 additional replica(s) M: 43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379 slots: (0 slots) slave replicates e621db96b845eb7818eb01a37fccdc7796f8d7de S: 510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379 slots: (0 slots) slave replicates 43906d49ffe4be7085b4d37f52c50ac7bb4df751 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
3、查看主从状态
bash
[root@localhost ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning info replication # Replication role:master connected_slaves:1 slave0:ip=192.168.112.50,port=6379,state=online,offset=630,lag=0 master_replid:54b3983a20bf4824a13d2699a9775fe3cb85ef0b master_replid2:0000000000000000000000000000000000000000 master_repl_offset:630 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:630 [root@localhost ~]# redis-cli -c -h 192.168.112.50 -a centos --no-auth-warning info replication # Replication role:slave master_host:192.168.112.10 master_port:6379 master_link_status:up master_last_io_seconds_ago:8 master_sync_in_progress:0 slave_repl_offset:756 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:54b3983a20bf4824a13d2699a9775fe3cb85ef0b master_replid2:0000000000000000000000000000000000000000 master_repl_offset:756 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:756
4、查看集群node对应关系
bash
[root@localhost ~]# redis-cli -a centos --no-auth-warning cluster nodes c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379@16379 myself,master - 0 1713536198000 1 connected 0-5460 bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379@16379 slave c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 0 1713536200000 5 connected e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379@16379 master - 0 1713536202879 3 connected 10923-16383 43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379@16379 master - 0 1713536201862 2 connected 5461-10922 5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379@16379 slave e621db96b845eb7818eb01a37fccdc7796f8d7de 0 1713536201000 3 connected 510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379@16379 slave 43906d49ffe4be7085b4d37f52c50ac7bb4df751 0 1713536200845 6 connected
5、验证集群状态
bash
[root@localhost ~]# redis-cli -a centos --no-auth-warning cluster info cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 #节点数 cluster_size:3 #3个集群 cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_ping_sent:895 cluster_stats_messages_pong_sent:791 cluster_stats_messages_sent:1686 cluster_stats_messages_ping_received:786 cluster_stats_messages_pong_received:895 cluster_stats_messages_meet_received:5 cluster_stats_messages_received:1686
6、模拟master故障
- 对应的slave节点自动提升为新master
- 模拟master1节点出故障,需要相应的数秒故障转移时间
master1关闭redis服务
bash
systemctl stop redis
master2上查看集群信息
bash
[root@localhost ~]# redis-cli -h 192.168.112.20 -a centos --no-auth-warning cluster nodes e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379@16379 master - 0 1713536909635 3 connected 10923-16383 510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379@16379 slave 43906d49ffe4be7085b4d37f52c50ac7bb4df751 0 1713536907592 2 connected 43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379@16379 myself,master - 0 1713536908000 2 connected 5461-10922 c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379@16379 slave,fail bb81813b153345f0406cfe88c1899e11e2e7a84f 1713536880567 1713536878000 7 disconnected bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379@16379 master - 0 1713536908615 7 connected 0-5460 5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379@16379 slave e621db96b845eb7818eb01a37fccdc7796f8d7de 0 1713536908000 4 connected
可以看到master1对应的slave3已经提升为master了