【学习笔记】redis 6.0.3 三主三从集群搭建

一、简介

Redis集群(Redis Cluster) 是Redis提供的分布式数据库方案，通过 分片(sharding) 来进行数据共享，并提供复制和故障转移功能。相比于主从复制、哨兵模式，Redis集群实现了较为完善的高可用方案，解决了存储能力受到单机限制，写操作无法负载均衡的问题。
三主三从集群模式搭建，环境基于centos7 + Redis6.0.3

本文记录学习redis集群的实践总结，逐步介绍搭建集群环境的过程。

二、环境搭建

1、Redis

redis使用的是6.0.3版本,安装教程可以看【学习笔记】Redis-搭建 centos7 安装redis 6.0.3

2、服务器

为了更改地贴近真实场景，三主三从模式部署在三台服务器上，服务器方面我这边依旧采用VMware16 pro创建centos7虚拟机，搭建三台，每台各起两个redis服务：

192.168.229.100:7290 192.168.229.100:7291
192.168.229.120:7290 192.168.229.120:7291
192.168.229.130:7290 192.168.229.130:7291

三、集群搭建

1、创建集群文件夹，存放不同配置文件

[root@root ~]# cd /usr/local
[root@root local]# mkdir redis-cluster
[root@root local]# cd redis-cluster
[root@root redis-cluster]# mkdir 7290 7291
[root@root redis-cluster]# ls
7290 7291

2、对7290进行配置

复制配置到7290文件夹底下

[root@root redis-cluster]# cd 7290
[root@root 7290]# cp /usr/local/redis/redis.conf .
[root@root 7290]# ls
redis.conf

修改7290配置

[root@root 7290]# vim redis.conf

port 7290 ## 7290端口
daemonize yes ## 后台启动
bind 192.168.221.150 ## 指定bind的IP
cluster-enabled yes ## 开启redis-cluster集群
cluster-config-file nodes_7290.conf ## 每个实例还包含存储此节点配置的文件的路径
cluster-node-timeout 5000 ## 超时时间
appendonly yes ## 开启aof

3、同步修改7291配置

复制7290的配置

[root@root 7290]# cp redis.conf ../7291
[root@root 7290]# sed -i 's/7290/7291/g' ../7291/redis.conf

4、启动redis

[root@root 7290]# cd /usr/local/bin
[root@root bin]# ls
redis-benchmark  redis-check-aof  redis-check-rdb  redis-cli  redis-sentinel  redis-server
[root@root bin]# ./redis-server /usr/local/redis-cluster/7290/redis.conf
[root@root bin]# ./redis-server /usr/local/redis-cluster/7291/redis.conf

查看启动状态

[root@root bin]# ps -ef|grep redis
root       1697      1  0 17:56 ?        00:00:00 ./redis-server 192.168.229.100:7290 [cluster]
root       1703      1  0 17:56 ?        00:00:00 ./redis-server 192.168.229.100:7291 [cluster]

5、以相同的方式部署其他两台服务器

6、开启集群总线端口

防火墙关闭的童鞋可忽略
这是我作为小白遇到的一个坑，在启动集群的时候还有端口没有开启导致一直wait

redis集群不仅需要开通redis客户端连接的端口，而且需要开通集群总线端口
集群总线端口为redis客户端连接的端口 + 10000

我们使用的是7290和7291端口，因此三台服务器都需要开启17290和17291端口

firewall-cmd --zone=public --add-port=17290/tcp --permanent
firewall-cmd --zone=public --add-port=17291/tcp --permanent
firewall-cmd --reload

四、集群启动

1、redis-cli创建集群

redis5.0之后，可以直接使用redis-cli直接创建集群，通过该方式创建的会自动分配主从节点，并且redis集群会尽量把主从服务器分配在不同机器上

[root@root bin]# ./redis-cli -a redis --cluster create 192.168.229.100:7290 192.168.229.100:7291 192.168.229.120:7290 192.168.229.120:7291 192.168.229.130:7290 192.168.229.130:7291 --cluster-replicas 1

集群参数解释：
-a redis：redis 是密码 根据各自情况配置
cluster-replicas 1：表示希望为集群中的每个主节点创建一个从节点(一主一从)
cluster-replicas 2：表示希望为集群中的每个主节点创建两个从节点(一主二从)
我们采用的是cluster-replicas 1 三主三从

如果出现以下报错，是因为服务器端口未开启，防火墙开启端口即可，如果怕麻烦就关闭防火墙（自己玩玩不用管）

[ERR] Node 192.168.229.100:7290 NOAUTH Authentication required

firewall-cmd --zone=public --add-port=7290/tcp --permanent
firewall-cmd --zone=public --add-port=7291/tcp --permanent
firewall-cmd --reload

开启后重新执行，以下是执行成功结果

[root@root bin]# ./redis-cli -a redis --cluster create 192.168.229.100:7290 192.168.229.100:7291 192.168.229.120:7290 192.168.229.120:7291 192.168.229.130:7290 192.168.229.130:7291 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.229.120:7291 to 192.168.229.100:7290
Adding replica 192.168.229.130:7291 to 192.168.229.120:7290
Adding replica 192.168.229.100:7291 to 192.168.229.130:7290
M: 1446a302ed93eb5d5f9ef22587ff63ecbaa56660 192.168.229.100:7290
   slots:[0-5460] (5461 slots) master
S: 281309a7a74f83937b4bdf7962ba89372ed54d9b 192.168.229.100:7291
   replicates 61e8850a0f0fc33cf1ed3a308c198ad2de915a47
M: bd489bd8e9bb2f9894889e2b0709b1f881f8f294 192.168.229.120:7290
   slots:[5461-10922] (5462 slots) master
S: 2287abe686cf00645509afaeed6d3941516b1842 192.168.229.120:7291
   replicates 1446a302ed93eb5d5f9ef22587ff63ecbaa56660
M: 61e8850a0f0fc33cf1ed3a308c198ad2de915a47 192.168.229.130:7290
   slots:[10923-16383] (5461 slots) master
S: c17bcaa2b613df2e3ec09792bceb0e4b7f9bab24 192.168.229.130:7291
   replicates bd489bd8e9bb2f9894889e2b0709b1f881f8f294
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 192.168.229.100:7290)
M: 1446a302ed93eb5d5f9ef22587ff63ecbaa56660 192.168.229.100:7290
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 281309a7a74f83937b4bdf7962ba89372ed54d9b 192.168.229.100:7291
   slots: (0 slots) slave
   replicates 61e8850a0f0fc33cf1ed3a308c198ad2de915a47
M: bd489bd8e9bb2f9894889e2b0709b1f881f8f294 192.168.229.120:7290
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 2287abe686cf00645509afaeed6d3941516b1842 192.168.229.120:7291
   slots: (0 slots) slave
   replicates 1446a302ed93eb5d5f9ef22587ff63ecbaa56660
M: 61e8850a0f0fc33cf1ed3a308c198ad2de915a47 192.168.229.130:7290
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: c17bcaa2b613df2e3ec09792bceb0e4b7f9bab24 192.168.229.130:7291
   slots: (0 slots) slave
   replicates bd489bd8e9bb2f9894889e2b0709b1f881f8f294
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

2、Waiting问题处理

如果忘了开放端口导致一直Waiting for the cluster to join…，那就先停止集群启动，先去开放端口（当然，防火墙都直接关闭了的就省略这个好吧）

开启后不能直接重启执行开启集群命令，不然会出现以下报错：

[ERR] Node 192.168.229.100:7290 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

解决办法：

1、先kill redis创建的集群节点进程
2、删除appendonly.aof、dump.rdb、nodes.conf文件，文件位置在数据存储目录：/usr/local/redis/data
3、执行./redis-cli flushdb 命令，清空每个redis里面的数据。
4、重启redis节点，再重新启动集群即可

五、集群操作

1、查看集群信息

选择一个节点进入

redis-cli -a redis -h ip -p 端口 -c

[root@root redis-cluster]# redis-cli -a redis -h 192.168.229.100 -p 7290 -c
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.229.100:7290> cluster nodes
281309a7a74f83937b4bdf7962ba89372ed54d9b 192.168.229.100:7291@17291 slave 61e8850a0f0fc33cf1ed3a308c198ad2de915a47 0 1648729497085 5 connected
bd489bd8e9bb2f9894889e2b0709b1f881f8f294 192.168.229.120:7290@17290 master - 0 1648729497085 3 connected 5461-10922
2287abe686cf00645509afaeed6d3941516b1842 192.168.229.120:7291@17291 slave 1446a302ed93eb5d5f9ef22587ff63ecbaa56660 0 1648729498094 4 connected
61e8850a0f0fc33cf1ed3a308c198ad2de915a47 192.168.229.130:7290@17290 master - 0 1648729499103 5 connected 10923-16383
c17bcaa2b613df2e3ec09792bceb0e4b7f9bab24 192.168.229.130:7291@17291 slave bd489bd8e9bb2f9894889e2b0709b1f881f8f294 0 1648729498094 6 connected
1446a302ed93eb5d5f9ef22587ff63ecbaa56660 192.168.229.100:7290@17290 myself,master - 0 1648729498000 1 connected 0-5460
192.168.229.100:7290> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:3367
cluster_stats_messages_pong_sent:3434
cluster_stats_messages_sent:6801
cluster_stats_messages_ping_received:3429
cluster_stats_messages_pong_received:3367
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:6801

2、数据验证操作

插入数据

192.168.229.100:7290> set testkey "Hello World"
OK
192.168.229.100:7290> get testkey
"Hello World"

从其他节点获取刚才插入的数据

[root@root bin]# redis-cli -a redis -h 192.168.229.120 -p 7291 -c
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.229.120:7291> get testkey
-> Redirected to slot [4757] located at 192.168.229.100:7290
"Hello World"

至此，集群搭建成功。