文章目录
一、redis cluster
Redis Cluster 工作原理
在哨兵sentinel机制中,可以解决redis高可用问题,即当master故障后可以自动将slave提升为master,从而可以保证redis服务的正常使用,但是无法解决redis单机写入的瓶颈问题,即单机redis写入性能受限于单机的内存大小、并发数量、网卡速率等因素。
- 早期Redis 分布式集群部署方案
- 客户端分区:由客户端程序决定key写分配和写入的redis node,但是需要客户端自己处理写入分配、高可用管理和故障转移等
- 代理方案:基于三方软件实现redis proxy,客户端先连接至代理层,由代理层实现key的写入分配,对客户端来说是有比较简单,但是对于集群管理节点增减相对比较麻烦,而且代理本身也是单点和性能瓶颈。
redis 3.0版本之后推出了无中心架构的redis cluster机制,在无中心的redis集群当中,其每个节点保存当前节点数据和整个集群状态,每个节点都和其他所有节点连接
- Redis Cluster特点如下
- 所有Redis节点使用(PING机制)互联
- 集群中某个节点的是否失效,是由整个集群中超过半数的节点监测都失效,才能算真正的失效
- 客户端不需要proxy即可直接连接redis,应用程序中需要配置有全部的redis服务器IP
- redis cluster把所有的redis node 平均映射到 0-16383个槽位(slot)上,读写需要到指定的redis node上进行操作,因此有多少个redis node相当于redis 并发扩展了多少倍,每个redisnode 承担16384/N个槽位
- Redis cluster预先分配16384个(slot)槽位,当需要在redis集群中写入一个key -value的时候,会使用CRC16(key) mod 16384之后的值,决定将key写入值哪一个槽位从而决定写入哪一个Redis节点上,从而有效解决单机瓶颈。
Redis cluster 基本架构
假如三个主节点分别是:A,B,C三个节点,采用哈希槽(hash slot)的方式来分配16384个slot的活,它们三个节点分别承担的slot 区间可以是:
- 节点A覆盖:0-5460
- 节点B覆盖:5461-10922
- 节点C覆盖:10923-16383
Redis cluster主从架构
Redis cluster的架构虽然解决了并发的问题,但是又引入了一个新的问题,每个Redis master的高可用如何解决?
那就是对每个master 节点都实现主从复制,从而实现 redis 高可用性
Redis Cluster 部署架构说明
- 环境A:3台服务器,每台服务器启动6379和6380两个redis 服务实例,适用于测试环境
- 环境B:6台服务器,分别是三组master/slave,适用于生产环境
部署方式介绍
-
redis cluster 有多种部署方法
-
原生命令安装
- 理解Redis Cluster架构
- 生产环境不使用
-
官方工具安装
- 高效、准确
- 生产环境可以使用
- 5.0版本后才支持
-
自主研发
- 可以实现可视化的自动化部署
原生命令手动部署
- 在所有节点安装redis,并配置开启cluster功能
- 各个节点执行meet,实现所有节点的相互通信
- 为各个master 节点指派槽位范围
- 指定各个节点的主从关系
原生命令
- 集群
- cluster info :打印集群的信息
- cluster nodes :列出集群当前已知的所有节点( node),以及这些节点的相关信息。
- 节点
- cluster meet :将 ip 和 port 所指定的节点添加到集群当中,让它成为集群的一份子。
- cluster forget <node_id> :从集群中移除 node_id 指定的节点。
- cluster replicate <master_node_id> :将当前从节点设置为 node_id 指定的master节点的slave节点。只能针对slave节点操作。
- cluster saveconfig :将节点的配置文件保存到硬盘里面。
- 槽(slot)
- cluster addslots [slot …] :将一个或多个槽( slot)指派( assign)给当前节点。
- cluster delslots [slot …] :移除一个或多个槽对当前节点的指派。
- cluster flushslots :移除指派给当前节点的所有槽,让当前节点变成一个没有指派任何槽的节点。
- cluster setslot node <node_id> :将槽 slot 指派给 node_id 指定的节点,如果槽已经指派给
- 另一个节点,那么先让另一个节点删除该槽>,然后再进行指派。
- cluster setslot migrating <node_id> :将本节点的槽 slot 迁移到 node_id 指定的节点中。
- cluster setslot importing <node_id> :从 node_id 指定的节点中导入槽 slot 到本节点。
- cluster setslot stable :取消对槽 slot 的导入( import)或者迁移( migrate)。
- 键
- cluster keyslot :计算键 key 应该被放置在哪个槽上。
- cluster countkeysinslot :返回槽 slot 目前包含的键值对数量。
- cluster getkeysinslot :返回 count 个 slot 槽中的键 。
实战案例:利用原生命令手动部署redis cluster
- 准备6台虚拟机
- 在所有节点安装redis并启动cluster功能
#在克隆之前将如下操作在模板上操作完成
[root@localhost ~]# vim redis_install.sh
#参照一键编译安装redis脚本
[root@localhost ~]# sh redis_install.sh
# 开始编译安装redis
- 将克隆过的主机的IP地址配置完成,防火墙放行对应的端口号
firewall-cmd --add-port=1-65535/tcp --permanent
firewall-cmd --add-port=1-65535/udp --permanent
firewall-cmd --reload
hostnamectl set-hostname master1
- 所有节点启用redis集群支持
echo "cluster-enabled yes" >> /apps/redis/etc/redis.conf
echo "masterauth centos" >> /apps/redis/etc/redis.conf
[root@master1 ~]# vim /apps/redis/etc/redis.conf
cluster-enabled yes
masterauth centos
[root@master1 ~]# systemctl restart redis
- 执行 meet 操作实现相互通信
# 在任意一节点上和其它所有节点进行meet通信,以master1为例
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster meet 192.168.175.11 6379
OK
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster meet 192.168.175.20 6379
OK
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster meet 192.168.175.21 6379
OK
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster meet 192.168.175.30 6379
OK
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster meet 192.168.175.31 6379
OK
#可以看到所有节点之间可以相互连接通信,以master1为例
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster nodes
76da144ef9ea65664f9ba5ac431ed9e26b07b132 192.168.175.31:6379@16379 master - 0 1625909926530 4 connected
fae9fe337678873ec0f3e0a44c39bd8064c85fb0 192.168.175.30:6379@16379 master - 0 1625909925000 0 connected
5eeb5ce103710ec5108d3257fdcd019b401548d9 192.168.175.11:6379@16379 master - 0 1625909926000 3 connected
3ccb32a3e79572a16a1aa3e8188fff07121b1d1e 192.168.175.20:6379@16379 master - 0 1625909927549 2 connected
48000fa51e653b3dbf3f70829fc9142c40195ff6 192.168.175.10:6379@16379 myself,master - 0 1625909927000 1 connected
e1b815df722f7c202e28215fa23ff20913d88479 192.168.175.21:6379@16379 master - 0 1625909924486 5 connected
- 由于没有分配槽位,无法创建key
[root@master1 ~]# redis-cli -a centos --no-auth-warning set name eagle
(error) CLUSTERDOWN Hash slot not served
- 查看当前状态,以master1为例
[root@master1 ~]# redis-cli -a centos --no-auth-warning cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:289
cluster_stats_messages_pong_sent:290
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:584
cluster_stats_messages_ping_received:289
cluster_stats_messages_pong_received:294
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:584
- 为每个master 节点指派槽位范围
[root@master1 ~]# vim addslots.sh
#!/bin/bash
HOST=$1
PORT=$2
START=$3
END=$4
PASS=centos
for slot in `seq ${START} ${END}`;do
echo "slot: ${slot}"
redis-cli -h ${HOST} -p ${PORT} -a ${PASS} --no-auth-warning cluster addslots ${slot}
done
* 为三个master分配槽位,共16364/3=5461.33333,平均每个master分配5461个槽位
[root@master1 ~]# bash addslots.sh 192.168.175.10 6379 0 5461
#当master1分配完槽位后,可以看到下面信息
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster info
cluster_state:fail
cluster_slots_assigned:5462
cluster_slots_ok:5462
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:1
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:504
cluster_stats_messages_pong_sent:500
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:1009
cluster_stats_messages_ping_received:499
cluster_stats_messages_pong_received:509
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:1009
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster nodes
76da144ef9ea65664f9ba5ac431ed9e26b07b132 192.168.175.31:6379@16379 master - 0 1625910366128 4 connected
fae9fe337678873ec0f3e0a44c39bd8064c85fb0 192.168.175.30:6379@16379 master - 0 1625910366000 0 connected
5eeb5ce103710ec5108d3257fdcd019b401548d9 192.168.175.11:6379@16379 master - 0 1625910365000 3 connected
3ccb32a3e79572a16a1aa3e8188fff07121b1d1e 192.168.175.20:6379@16379 master - 0 1625910367145 2 connected
48000fa51e653b3dbf3f70829fc9142c40195ff6 192.168.175.10:6379@16379 myself,master - 0 1625910366000 1 connected 0-5461
e1b815df722f7c202e28215fa23ff20913d88479 192.168.175.21:6379@16379 master - 0 1625910368168 5 connected
[root@master1 ~]# bash addslots.sh 192.168.175.20 6379 5462 10922
[root@master1 ~]# bash addslots.sh 192.168.175.30 6379 10923 16383
# 当所有的三个master分配完槽位后,可以看到如下信息,以master1为例,在其它主机看到的信息一样
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster nodes
76da144ef9ea65664f9ba5ac431ed9e26b07b132 192.168.175.31:6379@16379 master - 0 1625912355000 4 connected
3ccb32a3e79572a16a1aa3e8188fff07121b1d1e 192.168.175.20:6379@16379 master - 0 1625912353000 2 connected 5462-10922
fae9fe337678873ec0f3e0a44c39bd8064c85fb0 192.168.175.30:6379@16379 master - 0 1625912355795 0 connected 10923-16383
5eeb5ce103710ec5108d3257fdcd019b401548d9 192.168.175.11:6379@16379 master - 0 1625912354000 3 connected
48000fa51e653b3dbf3f70829fc9142c40195ff6 192.168.175.10:6379@16379 myself,master - 0 1625912351000 1 connected 0-5461
e1b815df722f7c202e28215fa23ff20913d88479 192.168.175.21:6379@16379 master - 0 1625912354000 5 connected
- 指定各个节点的主从关系
# 通过上面的cluster nodes 查看master的ID信息,执行下面操作,将对应的slave 指定相应的master节点,实现三对主从节点
[root@master1 ~]# redis-cli -h 192.168.175.11 -a centos --no-auth-warning cluster replicate 48000fa51e653b3dbf3f70829fc9142c40195ff6
OK
[root@master1 ~]# redis-cli -h 192.168.175.21 -a centos --no-auth-warning cluster replicate 3ccb32a3e79572a16a1aa3e8188fff07121b1d1e
OK
[root@master1 ~]# redis-cli -h 192.168.175.31 -a centos --no-auth-warning cluster replicate fae9fe337678873ec0f3e0a44c39bd8064c85fb0
OK
- 查看主从节点关系及槽位信息
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning cluster slots
1) 1) (integer) 5462
2) (integer) 10922
3) 1) "192.168.175.20"
2) (integer) 6379
3) "3ccb32a3e79572a16a1aa3e8188fff07121b1d1e"
4) 1) "192.168.175.21"
2) (integer) 6379
3) "e1b815df722f7c202e28215fa23ff20913d88479"
2) 1) (integer) 10923
2) (integer) 16383
3) 1) "192.168.175.30"
2) (integer) 6379
3) "fae9fe337678873ec0f3e0a44c39bd8064c85fb0"
4) 1) "192.168.175.31"
2) (integer) 6379
3) "76da144ef9ea65664f9ba5ac431ed9e26b07b132"
3) 1) (integer) 0
2) (integer) 5461
3) 1) "192.168.175.10"
2) (integer) 6379
3) "48000fa51e653b3dbf3f70829fc9142c40195ff6"
4) 1) "192.168.175.11"
2) (integer) 6379
3) "5eeb5ce103710ec5108d3257fdcd019b401548d9"
- 验证 redis cluster 访问
- -c 表示以集群方式连接
[root@master1 ~]# redis-cli -c -h 192.168.175.10 -a centos --no-auth-warning set name eagle
OK
[root@master1 ~]# redis-cli -c -h 192.168.175.10 -a centos --no-auth-warning get name
"eagle"
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning get name
(error) MOVED 5798 192.168.175.20:6379
[root@master1 ~]# redis-cli -h 192.168.175.20 -a centos --no-auth-warning get name
"eagle"
[root@master1 ~]# redis-cli -h 192.168.175.30 -a centos --no-auth-warning get name
(error) MOVED 5798 192.168.175.20:6379
实战案例:基于Redis 5 的redis cluster部署
官方文档:https://redis.io/topics/cluster-tutorial
# cluster选项帮助
[root@master1 ~]# redis-cli --cluster help
Cluster Manager Commands:
create host1:port1 ... hostN:portN
--cluster-replicas <arg>
check host:port
--cluster-search-multiple-owners
info host:port
fix host:port
--cluster-search-multiple-owners
reshard host:port
--cluster-from <arg>
--cluster-to <arg>
--cluster-slots <arg>
--cluster-yes
--cluster-timeout <arg>
--cluster-pipeline <arg>
--cluster-replace
rebalance host:port
--cluster-weight <node1=w1...nodeN=wN>
--cluster-use-empty-masters
--cluster-timeout <arg>
--cluster-simulate
--cluster-pipeline <arg>
--cluster-threshold <arg>
--cluster-replace
add-node new_host:new_port existing_host:existing_port
--cluster-slave
--cluster-master-id <arg>
del-node host:port node_id
call host:port command arg arg .. arg
set-timeout host:port milliseconds
import host:port
--cluster-from <arg>
--cluster-copy
--cluster-replace
help
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
- 创建 redis cluster集群的环境准备
- 每个redis 节点采用相同的硬件配置、相同的密码、相同的redis版本
- 所有redis服务器必须没有任何数据
- 准备6台主机,开启cluster配置
创建集群
#哨兵配置
echo "masterauth centos" >> /apps/redis/etc/redis.conf
echo "cluster-enabled yes" >> /apps/redis/etc/redis.conf
#集群开启
systemctl restart redis
[root@master1 ~]# redis-cli -a centos --no-auth-warning --cluster create \
192.168.175.10:6379 192.168.175.20:6379 192.168.175.30:6379 \
192.168.175.11:6379 192.168.175.21:6379 192.168.175.31:6379 \
--cluster-replicas 1
# redis-cli --cluster-replicas 1 表示每个master对应一个slave节点
# 观察运行结果,可以看到3组master/slave
查看主从状态
[root@master1 ~]# redis-cli -c -h 192.168.175.10 -a centos info replication
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:0
master_replid:19f199fca5cf6fcd15aa6db3d7482c302923f84d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
[root@master1 ~]# redis-cli -c -h 192.168.175.11 -a centos info replication
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:slave
master_host:192.168.175.30
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:1
master_link_down_since_seconds:1625915270
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:d322eda9ed8308deae297ad5920a22e43a1d24ab
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
- 查看指定master节点的slave节点信息
[root@master1 ~]# redis-cli -a centos cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
449ef170cd83b0d42c47e6504210699770b7a9fa 192.168.175.30:6379@16379 master - 0 1625915361650 3 connected 10923-16383
1329d448ec1f4abcf39ba03cdeed484c869a27b5 192.168.175.10:6379@16379 myself,master - 0 1625915359000 1 connected 0-5460
464d5cbe572a69ae6c95c82f2ce5942dbc8a848a 192.168.175.31:6379@16379 slave f8f0d5dec65001f4082c6db805fff98777636e00 0 1625915361000 6 connected
09f2f31bf45b91a04bab1e798a3f053104edfcd5 192.168.175.21:6379@16379 slave 1329d448ec1f4abcf39ba03cdeed484c869a27b5 0 1625915362668 5 connected
74a4f661379361a38f4bef603bc1807025527b4b 192.168.175.11:6379@16379 slave 449ef170cd83b0d42c47e6504210699770b7a9fa 0 1625915361000 4 connected
f8f0d5dec65001f4082c6db805fff98777636e00 192.168.175.20:6379@16379 master - 0 1625915362000 2 connected 5461-10922
验证集群状态
[root@master1 ~]# redis-cli -a centos --no-auth-warning cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6 #节点数
cluster_size:3 #3个集群
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:342
cluster_stats_messages_pong_sent:362
cluster_stats_messages_sent:704
cluster_stats_messages_ping_received:357
cluster_stats_messages_pong_received:342
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:704
查看集群node对应关系
[root@master1 ~]# redis-cli -a centos --no-auth-warning cluster nodes
449ef170cd83b0d42c47e6504210699770b7a9fa 192.168.175.30:6379@16379 master - 0 1625915444000 3 connected 10923-16383
1329d448ec1f4abcf39ba03cdeed484c869a27b5 192.168.175.10:6379@16379 myself,master - 0 1625915446000 1 connected 0-5460
464d5cbe572a69ae6c95c82f2ce5942dbc8a848a 192.168.175.31:6379@16379 slave f8f0d5dec65001f4082c6db805fff98777636e00 0 1625915446000 6 connected
09f2f31bf45b91a04bab1e798a3f053104edfcd5 192.168.175.21:6379@16379 slave 1329d448ec1f4abcf39ba03cdeed484c869a27b5 0 1625915447384 5 connected
74a4f661379361a38f4bef603bc1807025527b4b 192.168.175.11:6379@16379 slave 449ef170cd83b0d42c47e6504210699770b7a9fa 0 1625915443298 4 connected
f8f0d5dec65001f4082c6db805fff98777636e00 192.168.175.20:6379@16379 master - 0 1625915447000 2 connected 5461-10922
验证集群写入key
- redis cluster 写入key
[root@master1 ~]# redis-cli -h 192.168.175.10 -a centos --no-auth-warning set name eagle
(error) MOVED 5798 192.168.175.20:6379 #槽位不在当前node所以无法写入
[root@master1 ~]# redis-cli -h 192.168.175.20 -a centos --no-auth-warning set name eagle
OK
[root@master1 ~]# redis-cli -h 192.168.175.20 -a centos --no-auth-warning get name
"eagle"
模拟master故障
- 对应的slave节点自动提升为新master
- 模拟master3节点出故障,需要相应的数秒故障转移时间
[root@master3 ~]# redis-cli -a centos --no-auth-warning
127.0.0.1:6379> shutdown
not connected> exit
[root@master3 ~]# redis-cli -a centos --no-auth-warning --cluster info 192.168.175.10:6379
Could not connect to Redis at 192.168.175.30:6379: Connection refused
192.168.175.10:6379 (c5fd0661...) -> 3333 keys | 5461 slots | 1 slaves.
192.168.175.20:6379 (1cd30e11...) -> 3341 keys | 5462 slots | 1 slaves.
192.168.175.31:6379 (4e42a7da...) -> 3330 keys | 5461 slots | 0 slaves. #192.168.175.31为新的master
[OK] 10004 keys in 3 masters.
0.61 keys per slot on average.