redis三主三从
创建六个容器实例
#--net host 使用宿主机的ip和端口,默认
#--cluster-enabled yes 开启redis集群
#--appendonly yes 开启持久化
docker run -d --name=redis-node-1 --net host --privileged=true -v /mydata/redis/share/redis-node-1:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6381
docker run -d --name=redis-node-2 --net host --privileged=true -v /mydata/redis/share/redis-node-2:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6382
docker run -d --name=redis-node-3 --net host --privileged=true -v /mydata/redis/share/redis-node-3:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6383
docker run -d --name=redis-node-4 --net host --privileged=true -v /mydata/redis/share/redis-node-4:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6384
docker run -d --name=redis-node-5 --net host --privileged=true -v /mydata/redis/share/redis-node-5:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6385
docker run -d --name=redis-node-6 --net host --privileged=true -v /mydata/redis/share/redis-node-6:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6386
2.随便进入一台redis执行下边命令
#--cluster-replicas 1 表示为每一个master创建一个slave节点
redis-cli --cluster create xx.xx.xx.xx:6381 xx.xx.xx.xx:6382 xx.xx.xx.xx:6383 xx.xx.xx.xx:6384 xx.xx.xx.xx:6385 xx.xx.xx.xx:6386 --cluster-replicas 1
执行过命令出现以下内容就说明ok了
谁是master谁是slave,根据自己的实际情况来看
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica xx.xx.xx.xx:6385 to xx.xx.xx.xx:6381
Adding replica xx.xx.xx.xx:6386 to xx.xx.xx.xx:6382
Adding replica xx.xx.xx.xx:6384 to xx.xx.xx.xx:6383
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 8bce48eafe69b0ea553d4ea08d1cc67d5eee9539 xx.xx.xx.xx:6381
slots:[0-5460] (5461 slots) master
M: 88ab82fd932d48f837630ec740a46e39c1551607 xx.xx.xx.xx:6382
slots:[5461-10922] (5462 slots) master
M: 6e1879f230849d09217223d4ba7ccb14493204c7 xx.xx.xx.xx:6383
slots:[10923-16383] (5461 slots) master
S: 69991718b5a2650ce32e1cc180e1c9f7669c3699 xx.xx.xx.xx:6384
replicates 8bce48eafe69b0ea553d4ea08d1cc67d5eee9539
S: 4731f4c2239f40ad687ae933233c148f74b42c5f xx.xx.xx.xx:6385
replicates 88ab82fd932d48f837630ec740a46e39c1551607
S: 72eb9cf2e723c6f08d9b8152c39142e0f730791c xx.xx.xx.xx:6386
replicates 6e1879f230849d09217223d4ba7ccb14493204c7
Can I set the above configuration? (type 'yes' to accept):
之后输入yes等待加载
这里我犯了个错误,只开放了redis端口,没有开放集群总线端口,导致等了半个小时一直在loading,在这里我提出来,避免入坑
redis集群不仅需要开通redis客户端连接的端口,而且需要开通集群总线端口
集群总线端口为redis客户端连接的端口 + 10000
如redis端口为6379
则集群总线端口为16379
故,所有服务器的点需要开通redis的客户端连接端口和集群总线端口
3.进入容器,依次执行以下命令
docker exec -it redis-node-1 /bin/bash
redis-cli -p 6381
查看节点信息
cluster info
cluster_state:ok
#已分配的槽位
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
#已知节点
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:843
cluster_stats_messages_pong_sent:890
cluster_stats_messages_sent:1733
cluster_stats_messages_ping_received:885
cluster_stats_messages_pong_received:843
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1733
cluster nodes
4731f4c2239f40ad687ae933233c148f74b42c5f xx.xx.xx.xx:6385@16385 slave 8bce48eafe69b0ea553d4ea08d1cc67d5eee9539 0 1649088809869 1 connected
8bce48eafe69b0ea553d4ea08d1cc67d5eee9539 xx.xx.xx.xx:6381@16381 myself,master - 0 1649088807000 1 connected 0-5460
6e1879f230849d09217223d4ba7ccb14493204c7 xx.xx.xx.xx:6383@16383 master - 0 1649088807865 3 connected 10923-16383
69991718b5a2650ce32e1cc180e1c9f7669c3699 xx.xx.xx.xx:6384@16384 slave 6e1879f230849d09217223d4ba7ccb14493204c7 0 1649088809000 3 connected
72eb9cf2e723c6f08d9b8152c39142e0f730791c xx.xx.xx.xx:6386@16386 slave 88ab82fd932d48f837630ec740a46e39c1551607 0 1649088808000 2 connected
88ab82fd932d48f837630ec740a46e39c1551607 xx.xx.xx.xx:6382@16382 master - 0 1649088807000 2 connected 5461-10922
可以看得到那几个是master,哪几个是slave,以及谁和谁绑定
4.连接方式
我们配置好以上操作,这时候就不能用redis-cli -p 6381的方式连接redis客户端了,由于集群会分配一个槽位,当进入node-1,往里面set值时,会有以下情况
#单机版链接
root@iZwz95fxdd6v2zhn5ifch9Z:/data# redis-cli -p 6381
127.0.0.1:6381> set k1 v1
(error) MOVED 12706 xx.xx.xx.xx:6383
127.0.0.1:6381> set k2 v2
OK
127.0.0.1:6381> set k3 v3
OK
127.0.0.1:6381> set k4 v4
(error) MOVED 8455 xx.xx.xx.xx:6382
127.0.0.1:6381>
发现有的可以set进去,而有的就会报错,报错也很明显,让我们将12706移到106.177.213:6383去进行存储
这是因为我们现在集群已经把这六台分成了三段,一主一从为一段,不同的key有不同的槽位,而槽位也是根据hash算法计算出来的
6381-6385 | 6382-6386 | 6383-6384 | 三部分 |
---|---|---|---|
0-16383个槽位 | 三部分共有的槽位 | ||
0-5460 | 5461-10922 | 10923-16383 | 将共有槽位分为三部分 |
这时候上述错误就很明白了,说的是将12706移动到xx.xx.xx.xx:6383存储,也就是移动到6383-6384进行存储
所以这时候要用的是用集群方式连接
#集群方式连接
root@iZwz95fxdd6v2zhn5ifch9Z:/data# redis-cli -p 6381 -c
127.0.0.1:6381> FLUSHALL
OK
127.0.0.1:6381> exit
root@iZwz95fxdd6v2zhn5ifch9Z:/data# redis-cli -p 6381 -c
127.0.0.1:6381> set k1 v1
-> Redirected to slot [12706] located at xx.xx.xx.xx:6383
OK
xx.xx.xx.xx:6383> set k2 v2
-> Redirected to slot [449] located at xx.xx.xx.xx:6381
OK
xx.xx.xx.xx:6381> set k3 v3
OK
xx.xx.xx.xx:6381> set k4 v4
-> Redirected to slot [8455] located at xx.xx.xx.xx:6382
OK
xx.xx.xx.xx:6382>
这时候就不会报错了,它会自动重定向到对应的部分
退出连接,再次用下方方式检查集群
root@iZwz95fxdd6v2zhn5ifch9Z:/data# redis-cli --cluster check xx.xx.xx.xx:6381
xx.xx.xx.xx:6381 (8bce48ea...) -> 2 keys | 5461 slots | 1 slaves.
xx.xx.xx.xx:6383 (6e1879f2...) -> 1 keys | 5461 slots | 1 slaves.
xx.xx.xx.xx:6382 (88ab82fd...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 4 keys in 3 masters.
以上信息告诉我们每部分有多少个值
5.主从切换
场景:当master1宕机了,master1下的从库会不会继承master1的数据
操作:
1.停掉6381这个实例,来模拟6381宕机
2.进入6382查看状态,进入方式用集群方式进入redis-cli -p 6382 -c
3.cluster nodes,会出现以下信息
127.0.0.1:6382> cluster nodes
#注意看这里,刚才我们停掉了6381,有个fail表示该master已经不能用了
8bce48eafe69b0ea553d4ea08d1cc67d5eee9539 xx.xx.xx.xx:6381@16381 master,fail - 1649135662460 1649135659457 1 disconnected
#我们上边也看到6381master下绑定的是6385,6381宕机了,现在6385来顶替6381
4731f4c2239f40ad687ae933233c148f74b42c5f xx.xx.xx.xx:6385@16385 master - 0 1649135745000 7 connected 0-5460
88ab82fd932d48f837630ec740a46e39c1551607 xx.xx.xx.xx:6382@16382 myself,master - 0 1649135747000 2 connected 5461-10922
69991718b5a2650ce32e1cc180e1c9f7669c3699 xx.xx.xx.xx:6384@16384 slave 6e1879f230849d09217223d4ba7ccb14493204c7 0 1649135746692 3 connected
6e1879f230849d09217223d4ba7ccb14493204c7 xx.xx.xx.xx:6383@16383 master - 0 1649135746000 3 connected 10923-16383
72eb9cf2e723c6f08d9b8152c39142e0f730791c xx.xx.xx.xx:6386@16386 slave 88ab82fd932d48f837630ec740a46e39c1551607 0 1649135747695 2 connected
刚才的操作中我们存储的数据都是到master中的,现在6381master挂了,6385上位了,现在我们取刚才存入的值能否去得到
127.0.0.1:6382> get k1
-> Redirected to slot [12706] located at xx.xx.xx.xx:6383
"v1"
xx.xx.xx.xx:6383> get k2
-> Redirected to slot [449] located at xx.xx.xx.xx:6385
"v2"
xx.xx.xx.xx:6385> get k3
"v3"
xx.xx.xx.xx:6385> get k4
-> Redirected to slot [8455] located at xx.xx.xx.xx:6382
"v4"
xx.xx.xx.xx:6382>
可以看到,master挂了,slave上位成为master,并不影戏我们的取值
主从恢复:还有一个问题,当我们的6381恢复了,理应6381继续作为master,其实不然,就算6381恢复了6385还是master,这是我们要做的就是把6381恢复成master,其实这个很简单,我们只需要将6385停掉,那么已经恢复的6381就会重新成为master,等一小会儿再启动6385,这时候6385就会再次成为slave
6.主从扩容
现在有一个场景,三主三从已经满足不了需求了,现在需要再集群中扩容添加6387(master),6388(slave),由于我们使用的是hash槽,那么槽位是从新分配呢还是各个master匀出来一些槽位给新来的呢
两个问题,一个是新增机器,一个是槽位分配
新增两个实例
docker run -d --name=redis-node-7 --net host --privileged=true -v /mydata/redis/share/redis-node-7:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6387
docker run -d --name=redis-node-8 --net host --privileged=true -v /mydata/redis/share/redis-node-8:/data redis:6.0.8 --cluster-enabled yes --appendonly yes --port 6388
进入6387,添加到集群中
docker exec -it redis-node-7 /bin/bash
#add-node xx.xx.xx.xx:6387 xx.xx.xx.xx:6381 添加新节点xx.xx.xx.xx:6387到xx.xx.xx.xx:6381
redis-cli --cluster add-node xx.xx.xx.xx:6387 xx.xx.xx.xx:6381
redis-cli --cluster check xx.xx.xx.xx:6381
xx.xx.xx.xx:6381 (8bce48ea...) -> 2 keys | 5461 slots | 1 slaves.
#新加入的还没有槽位
xx.xx.xx.xx:6387 (a9aa95a4...) -> 0 keys | 0 slots | 0 slaves.
xx.xx.xx.xx:6382 (88ab82fd...) -> 1 keys | 5462 slots | 1 slaves.
xx.xx.xx.xx:6383 (6e1879f2...) -> 1 keys | 5461 slots | 1 slaves.
接下来重新分配槽位
redis-cli --cluster reshard xx.xx.xx.xx:6381
#接下来会有提示,按实际输入
#这个按照16384/master个数获得的值填写,我这变四台master,值就该是4096,注意不包括slave
How many slots do you want to move from (1 to 16384)?4096
#这个是说谁来接受分配的槽位,根据自己实际情况填写,是个很长的字符串a9aa95a4a13c98c8a71ddc35c93e80905331de51 对应的就是6387
What is the receiving node ID?a9aa95a4a13c98c8a71ddc35c93e80905331de51
#这里填写all
Source node #1: all
#之后yes就行了,等待慢慢分配
分配完成后我们再看下
redis-cli --cluster check xx.xx.xx.xx:6381
xx.xx.xx.xx:6381 (8bce48ea...) -> 1 keys | 4096 slots | 1 slaves.
#发现已经有了槽位
xx.xx.xx.xx:6387 (a9aa95a4...) -> 1 keys | 4096 slots | 0 slaves.
xx.xx.xx.xx:6382 (88ab82fd...) -> 1 keys | 4096 slots | 1 slaves.
xx.xx.xx.xx:6383 (6e1879f2...) -> 1 keys | 4096 slots | 1 slaves.
这个分配其实是之前三个master的槽位平均分了点槽位给新的master
将6388slave分配到6387master下
#add-node xx.xx.xx.xx:6388 xx.xx.xx.xx:6387 在6387下添加新的节点6388作为slave
#cluster-master-id a9aa95a4a13c98c8a71ddc35c93e80905331de51 master的id
redis-cli --cluster add-node xx.xx.xx.xx:6388 xx.xx.xx.xx:6387 --cluster-slave --cluster-master-id a9aa95a4a13c98c8a71ddc35c93e80905331de51
7.主从缩容
场景:和主从扩容相反,现在要去掉一主一从
我们要先删slave再删master
redis-cli --cluster check xx.xx.xx.xx:6381
xx.xx.xx.xx:6381 (8bce48ea...) -> 1 keys | 4096 slots | 1 slaves.
xx.xx.xx.xx:6387 (a9aa95a4...) -> 1 keys | 4096 slots | 1 slaves.
xx.xx.xx.xx:6382 (88ab82fd...) -> 1 keys | 4096 slots | 1 slaves.
xx.xx.xx.xx:6383 (6e1879f2...) -> 1 keys | 4096 slots | 1 slaves.
[OK] 4 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node xx.xx.xx.xx:6381)
M: 8bce48eafe69b0ea553d4ea08d1cc67d5eee9539 xx.xx.xx.xx:6381
slots:[1365-5460] (4096 slots) master
1 additional replica(s)
M: a9aa95a4a13c98c8a71ddc35c93e80905331de51 xx.xx.xx.xx:6387
slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
1 additional replica(s)
M: 88ab82fd932d48f837630ec740a46e39c1551607 xx.xx.xx.xx:6382
slots:[6827-10922] (4096 slots) master
1 additional replica(s)
S: 69991718b5a2650ce32e1cc180e1c9f7669c3699 xx.xx.xx.xx:6384
slots: (0 slots) slave
replicates 6e1879f230849d09217223d4ba7ccb14493204c7
S: 4731f4c2239f40ad687ae933233c148f74b42c5f xx.xx.xx.xx:6385
slots: (0 slots) slave
replicates 8bce48eafe69b0ea553d4ea08d1cc67d5eee9539
#记住这个需要删除的slave id
S: 74d49286fa538e89286ecc6c39009e873cf3c576 xx.xx.xx.xx:6388
slots: (0 slots) slave
replicates a9aa95a4a13c98c8a71ddc35c93e80905331de51
M: 6e1879f230849d09217223d4ba7ccb14493204c7 xx.xx.xx.xx:6383
slots:[12288-16383] (4096 slots) master
1 additional replica(s)
S: 72eb9cf2e723c6f08d9b8152c39142e0f730791c xx.xx.xx.xx:6386
slots: (0 slots) slave
replicates 88ab82fd932d48f837630ec740a46e39c1551607
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
执行删除6388
#del-node xx.xx.xx.xx:6388 74d49286fa538e89286ecc6c39009e873cf3c576 删除节点6388,id是上一步保存的id
redis-cli --cluster del-node xx.xx.xx.xx:6388 74d49286fa538e89286ecc6c39009e873cf3c576
>>> Removing node 74d49286fa538e89286ecc6c39009e873cf3c576 from cluster xx.xx.xx.xx:6388
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
先不要删6387,先将槽位重新分配
redis-cli --cluster reshard xx.xx.xx.xx:6381
#接下来步骤和扩容分配一样,只不过像平均分配到三个master的话就多次执行,比如去掉的master有4096个槽位,那就4096/3,当然,除不尽可以选择哪个master多一个,哪个少一个,之后在重复两次,直到最后一次6387没有槽位,我这里直接把所有槽位全部分给6381
How many slots do you want to move (from 1 to 16384)? 4096
#6381master分配4096个
What is the receiving node ID? 8bce48eafe69b0ea553d4ea08d1cc67d5eee9539
#6387的id
Source node #1:a9aa95a4a13c98c8a71ddc35c93e80905331de51
Source node #2:done
之后删除6387
redis-cli --cluster del-node xx.xx.xx.xx:6387 a9aa95a4a13c98c8a71ddc35c93e80905331de51