之前我们演示过如何在docker中部署redis哨兵模式,本文讲解如何在docker部署redis cluster模式
Redis Cluster着重于提高并发量
。
群集至少需要3主3从,且每个实例使用不同的配置文件。
在redis-cluster架构中,redis-master节点一般用于接收读写,而redis-slave节点则一般只用于备份
, 其与对应的master拥有相同的slot集合,若某个redis-master意外失效,则再将其对应的slave进行升级为临时redis-master。
在redis的官方文档中,对redis-cluster架构上,有这样的说明:在cluster架构下,默认的,一般redis-master用于接收读写,而redis-slave则用于备份,当有请求是在向slave发起时,会直接重定向到对应key所在的master来处理
。 但如果不介意读取的是redis-cluster中有可能过期的数据并且对写请求不感兴趣时,则亦可通过readonly
命令,将slave设置成可读,然后通过slave获取相关的key,达到读写分离。具体可以参阅redis官方文档等相关内容
本文中我们也设定6个节点,3主3从模式。设定的端口为7001~7006。首先在home目录下创建一个文件夹为redis,然后在创建一个目录为redis-cluster,在redis-cluster中创建6个文件夹,6个文件里都含有conf和data两个文件夹,然后在conf文件夹新建redis.conf,内容如下
# redis端口 #7001文件夹对应7001 7002对应7002。。。
port 7001
# 关闭保护模式
protected-mode no #关闭protected-mode模式,此时外部网络可以直接访问
# 开启集群
cluster-enabled yes
# 集群节点配置
cluster-config-file nodes.conf
# 超时
cluster-node-timeout 5000
# 集群节点IP host模式为宿主机IP
cluster-announce-ip 宿主机IP
# 集群节点端口 7001 - 7006
cluster-announce-port 7001 #7001文件夹对应7001 7002对应7002。。。
cluster-announce-bus-port 17001 #7001文件夹对应17001 7002对应17002。。。
# 开启 appendonly 备份模式
appendonly yes
# 每秒钟备份
appendfsync everysec
# 对aof文件进行压缩时,是否执行同步操作
no-appendfsync-on-rewrite no
# 当目前aof文件大小超过上一次重写时的aof文件大小的100%时会再次进行重写
auto-aof-rewrite-percentage 100
# 重写前AOF文件的大小最小值 默认 64mb
auto-aof-rewrite-min-size 64mb
requirepass 密码 #redis设定的密码
masterauth 884127 ##设定的密码
6个redis.conf都设置完成后,回到redis目录中,新建docker-compose.yml,这里我们用docker-compose进行测试环境的docker编排。
version: '3.7'
services:
redis7001:
image: 'redis'
container_name: redis7001
restart: always
command:
["redis-server", "/usr/local/etc/redis/redis.conf"]
volumes:
- ./redis-cluster/7001/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./redis-cluster/7001/data:/data
ports:
- "7001:7001"
- "17001:17001"
environment:
# 设置时区为上海,否则时间会有问题
- TZ=Asia/Shanghai
redis7002:
image: 'redis'
container_name: redis7002
restart: always
command:
["redis-server", "/usr/local/etc/redis/redis.conf"]
volumes:
- ./redis-cluster/7002/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./redis-cluster/7002/data:/data
ports:
- "7002:7002"
- "17002:17002"
environment:
# 设置时区为上海,否则时间会有问题
- TZ=Asia/Shanghai
redis7003:
image: 'redis'
container_name: redis7003
restart: always
command:
["redis-server", "/usr/local/etc/redis/redis.conf"]
volumes:
- ./redis-cluster/7003/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./redis-cluster/7003/data:/data
ports:
- "7003:7003"
- "17003:17003"
environment:
# 设置时区为上海,否则时间会有问题
- TZ=Asia/Shanghai
redis7004:
image: 'redis'
container_name: redis7004
restart: always
command:
["redis-server", "/usr/local/etc/redis/redis.conf"]
volumes:
- ./redis-cluster/7004/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./redis-cluster/7004/data:/data
ports:
- "7004:7004"
- "17004:17004"
environment:
# 设置时区为上海,否则时间会有问题
- TZ=Asia/Shanghai
redis7005:
image: 'redis'
container_name: redis7005
restart: always
command:
["redis-server", "/usr/local/etc/redis/redis.conf"]
volumes:
- ./redis-cluster/7005/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./redis-cluster/7005/data:/data
ports:
- "7005:7005"
- "17005:17005"
environment:
# 设置时区为上海,否则时间会有问题
- TZ=Asia/Shanghai
redis7006:
image: 'redis'
container_name: redis7006
restart: always
command:
["redis-server", "/usr/local/etc/redis/redis.conf"]
volumes:
- ./redis-cluster/7006/conf/redis.conf:/usr/local/etc/redis/redis.conf
- ./redis-cluster/7006/data:/data
ports:
- "7006:7006"
- "17006:17006"
environment:
# 设置时区为上海,否则时间会有问题
- TZ=Asia/Shanghai
然后在redis目录中执行
[root@iZbp1bq6vb70qq3lbmjlh0Z redis]# docker-compose up -d
Creating network "redis_default" with the default driver
Creating redis7001 ... done
Creating redis7003 ... done
Creating redis7005 ... done
Creating redis7006 ... done
Creating redis7002 ... done
Creating redis7004 ... done
[root@iZbp1bq6vb70qq3lbmjlh0Z redis]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
55e24cd0f4d1 redis "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:7004->7004/tcp, :::7004->7004/tcp, 6379/tcp, 0.0.0.0:17004->17004/tcp, :::17004->17004/tcp redis7004
774eb9a83795 redis "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:7002->7002/tcp, :::7002->7002/tcp, 6379/tcp, 0.0.0.0:17002->17002/tcp, :::17002->17002/tcp redis7002
41aa5da92d3f redis "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:7005->7005/tcp, :::7005->7005/tcp, 6379/tcp, 0.0.0.0:17005->17005/tcp, :::17005->17005/tcp redis7005
310810eeb305 redis "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:7003->7003/tcp, :::7003->7003/tcp, 6379/tcp, 0.0.0.0:17003->17003/tcp, :::17003->17003/tcp redis7003
1c084aaa8490 redis "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:7006->7006/tcp, :::7006->7006/tcp, 6379/tcp, 0.0.0.0:17006->17006/tcp, :::17006->17006/tcp redis7006
7140f0109668 redis "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:7001->7001/tcp, :::7001->7001/tcp, 6379/tcp, 0.0.0.0:17001->17001/tcp, :::17001->17001/tcp redis7001
接下来我们利用redis-cli工具进行集群设置。
首先我们进入redis7001容器中,执行以下命令。
#密码就是redis.conf中设定的密码
docker exec -it redis7001 redis-cli -p 7001 -a 密码 --cluster create 服务器ip:7001 服务器ip:7002 服务器ip:7003 服务器ip:7004 服务器ip:7005 服务器ip:7006 --cluster-replicas 1
然后就会出现以下信息。
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 服务器ip:7005 to 服务器ip:7001
Adding replica 服务器ip:7006 to 服务器ip:7002
Adding replica 服务器ip:7004 to 服务器ip:7003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001
slots:[0-5460] (5461 slots) master
M: eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002
slots:[5461-10922] (5462 slots) master
M: d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003
slots:[10923-16383] (5461 slots) master
S: 5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004
replicates 47d1bb69797150e405ae6346c6be50dce994164c
S: b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005
replicates eab3edfab399b1a4d8a7d0232633a256adcac670
S: 4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006
replicates d205a3e1175573dfcc11f5da5781c4910af7c17b
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 服务器ip:7001)
M: 47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004
slots: (0 slots) slave
replicates 47d1bb69797150e405ae6346c6be50dce994164c
M: d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005
slots: (0 slots) slave
replicates eab3edfab399b1a4d8a7d0232633a256adcac670
S: 4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006
slots: (0 slots) slave
replicates d205a3e1175573dfcc11f5da5781c4910af7c17b
M: eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
集群测试
接下来进行一些集群的基本测试
1. 查看集群通信是否正常
redis7001主节点对它的副本节点redis7005进行ping操作。
docker exec -it redis7001 redis-cli -h 服务器ip -p 7005 -a 密码 ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG
2. 测试简单存储
redis7001主节点客户端操作redis7003主节点
docker docker exec -it redis7001 redis-cli -h 服务器ip -p 7003 -a 密码## 连接上redis7003
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
服务器ip:7003> set name admin
(error) MOVED 5798 服务器ip:7002
服务器ip:7003>
##由于Redis Cluster会根据key进行hash运算,然后将key分散到不同slots,name的hash运算结果在redis7002节点上的slots中。所以我们操作redis7003写操作,然后根据key运算,自动路由到7002。然而error提示无法路由?没关系,差一个 -c 参数而已。
#再次运行查看结果如下:
docker docker exec -it redis7001 redis-cli -h 服务器ip -p 7003 -a 密码 -c
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
服务器ip:7003> set name admin
-> Redirected to slot [5798] located at 服务器ip:7002
OK
服务器ip:7002> get name
"admin"
3. 查看集群状态
服务器ip:7002> cluster nodes
541e37fe6c4c1719305ae58236d48a5dcdc2cde5 服务器ip:7002@17002 myself,master - 0 1637763650000 2 connected 5461-10922
3d215963e0d33153ffcaaac42cfc4f1e32040c5f 服务器ip:7004@17004 slave 0077c19f7a8351a578e8902e64b86590be2bf3e7 0 1637763651000 1 connected
825f9cbf6e718d6d290ae106a364a5eb748bd316 服务器ip:7005@17005 slave 541e37fe6c4c1719305ae58236d48a5dcdc2cde5 0 1637763651624 2 connected
2151df09dff7aea50eb522ff92180c9a138e050e 服务器ip:7003@17003 master - 0 1637763650000 3 connected 10923-16383
9ff4c4e95da28b301d408a99d1bedd5d9a06a93a 服务器ip:7006@17006 slave 2151df09dff7aea50eb522ff92180c9a138e050e 0 1637763651726 3 connected
0077c19f7a8351a578e8902e64b86590be2bf3e7 服务器ip:7001@17001 master - 0 1637763652230 1 connected 0-5460
4. 测试读写分离
服务器ip:7002> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=服务器ip,port=7005,state=online,offset=589,lag=0
master_failover_state:no-failover
master_replid:007793c00ebc6c4378724d0c2ef8db558effd1e2
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:589
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:589
得到redis7002节点对应的从节点为redis7005,进入7005 获取刚刚在redis7002设定的key为name的值。
[root@iZbp1bq6vb70qq3lbmjlh0Z redis]# docker exec -it redis7001 redis-cli -h 服务器ip -p 7005 -a 密码
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
服务器ip :7005> get name
(error) MOVED 5798 服务器ip :7002
##发现读不到,原来在redis cluster中,如果你要在slave读取数据,那么需要带先执行 readonly 指令,然后 get key
服务器ip :7005> readonly
OK
服务器ip :7005> get name
"admin"
容灾演练
现在我们杀掉主节点redis7002,看从节点redis7005是否会接替它的位置。
redis7002未杀掉之前,查看节点状态。
服务器ip:7005> cluster nodes
4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006@17006 slave d205a3e1175573dfcc11f5da5781c4910af7c17b 0 1637807788000 3 connected
5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004@17004 slave 47d1bb69797150e405ae6346c6be50dce994164c 0 1637807788569 1 connected
eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002@17002 master - 0 1637807789875 2 connected 5461-10922
d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003@17003 master - 0 1637807788871 3 connected 10923-16383
b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005@17005 myself,slave eab3edfab399b1a4d8a7d0232633a256adcac670 0 1637807789000 2 connected
47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001@17001 master - 0 1637807788000 1 connected 0-5460
杀掉之后
服务器ip:7005> cluster nodes
4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006@17006 slave d205a3e1175573dfcc11f5da5781c4910af7c17b 0 1637808034000 3 connected
5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004@17004 slave 47d1bb69797150e405ae6346c6be50dce994164c 0 1637808034609 1 connected
eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002@17002 master,fail - 1637808010507 1637808008000 2 disconnected
d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003@17003 master - 0 1637808034911 3 connected 10923-16383
b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005@17005 myself,master - 0 1637808033000 7 connected 5461-10922
##看到7005节点变成master
47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001@17001 master - 0 1637808034000 1 connected 0-5460
redis7002节点恢复后 就会变成7005的从节点
服务器ip:7005> cluster nodes
4569f8e2a030bf3da89cadd29fc89d524340dc21 服务器ip:7006@17006 slave d205a3e1175573dfcc11f5da5781c4910af7c17b 0 1637808081174 3 connected
5a45319ff542d2058ce1a839a40b091ad80ea695 服务器ip:7004@17004 slave 47d1bb69797150e405ae6346c6be50dce994164c 0 1637808082179 1 connected
eab3edfab399b1a4d8a7d0232633a256adcac670 服务器ip:7002@17002 slave b54e2fc1ca06a6a1b2a24ade5059420d988e0081 0 1637808081000 7 connected
d205a3e1175573dfcc11f5da5781c4910af7c17b 服务器ip:7003@17003 master - 0 1637808082000 3 connected 10923-16383
b54e2fc1ca06a6a1b2a24ade5059420d988e0081 服务器ip:7005@17005 myself,master - 0 1637808080000 7 connected 5461-10922
47d1bb69797150e405ae6346c6be50dce994164c 服务器ip:7001@17001 master - 0 1637808082000 1 connected 0-5460