搭建【Redis-Cluster in CentOS7.x】

准备环境

• redis-standalone 安装请参考：《搭建【Redis in CentOS7.x】》

节点名	角色	ip	port
redis11	node01	172.16.4.221	6379
redis12	node02	172.16.4.222	6379
redis13	node03	172.16.4.223	6379
redis14	node04	172.16.4.224	6379
redis15	node05	172.16.4.225	6379
redis16	node06	172.16.4.226	6379

集群配置

修改配置

• 配置文件中cluster-enabled选项为yes，服务启动后会自动生成nodes.conf文件（由redis自动维护）
• 注意：cluster中不能使用 “repilicaof <masterip> <masterport>”

[root@redis16 redis]# vi redis.my.conf
...
################################ REDIS CLUSTER  ###############################

# Normal Redis instances can't be part of a Redis Cluster; only nodes that are
# started as cluster nodes can. In order to start a Redis instance as a
# cluster node enable the cluster support uncommenting the following:
#
cluster-enabled yes
# Every cluster node has a cluster configuration file. This file is not
# intended to be edited by hand. It is created and updated by Redis nodes.
# Every Redis Cluster node requires a different cluster configuration file.
# Make sure that instances running in the same system do not have
# overlapping cluster configuration file names.
#
# 默认为redis安装目录下nodes.conf文件
cluster-config-file /usr/local/redis/nodes.conf
...

查看状态

[root@redis11 redis]# ./bin/redis-cli 
127.0.0.1:6379> info cluster
# cluster_enabled为1表示集群启动正常，但目前各个节点还是完全独立状态 
cluster_enabled:1

配置集群

1. 安装ruby环境，请参考 《安装【Ruby in CentOS7.x】》。

2. 开启所有节点的 集群总线端口 16379 防火墙。

[root@demo04 ~]# ansible cluster -m command -a "firewall-cmd --zone=public --add-port=16379/tcp --permanent"
[root@demo04 ~]# ansible sentinel -m command -a "firewall-cmd --reload"

3. 初始化集群：使用redis提供的原生工具redis-trib.rb（如果是源码安装，默认是在 ./src 目录下）。
   注意：初始化集群是，database[0] 库中不能有数据。切初始化后只有database[0]可用。

# redis-trib.rb 官方提供的ruby集群管理工具(已被redis-cli取代)
# --replicas 1 表示1主1从；如果有6台机器则会选出6/2=3台主节点（集群至少要3台主节点）
# 所有节点ip:port都加上，用空格隔开即可。
[root@redis12 redis]# ./bin/redis-cli --cluster create 172.16.4.221:6379 172.16.4.222:6379 172.16.4.223:6379 172.16.4.224:6379 172.16.4.225:6379 172.16.4.226:6379 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.16.4.225:6379 to 172.16.4.221:6379
Adding replica 172.16.4.226:6379 to 172.16.4.222:6379
Adding replica 172.16.4.224:6379 to 172.16.4.223:6379
M: ced14cee3bd9d9fe295925e83e18bb4760fd803a 172.16.4.221:6379
   slots:[0-5460] (5461 slots) master
M: 43f29b6becef4d5e07d258b0644ff5affee2f6de 172.16.4.222:6379
   slots:[5249],[5461-10922] (5462 slots) master
M: 9ec9ad9d79b50471f1821b9dd4e8be9c90ceac94 172.16.4.223:6379
   slots:[5249],[10923-16383] (5461 slots) master
S: 246e69ad920c6c3cf1e152b126deb697b969c611 172.16.4.224:6379
   replicates 9ec9ad9d79b50471f1821b9dd4e8be9c90ceac94
S: d76014a45f6a8d4b64babf07ec6858c1b1e48093 172.16.4.225:6379
   replicates ced14cee3bd9d9fe295925e83e18bb4760fd803a
S: a9790214d027a76817c89239c003ce42ac3a5e07 172.16.4.226:6379
   replicates 43f29b6becef4d5e07d258b0644ff5affee2f6de
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 172.16.4.221:6379)
S: ced14cee3bd9d9fe295925e83e18bb4760fd803a 172.16.4.221:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
S: 9ec9ad9d79b50471f1821b9dd4e8be9c90ceac94 172.16.4.223:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
S: 246e69ad920c6c3cf1e152b126deb697b969c611 172.16.4.224:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
M: a9790214d027a76817c89239c003ce42ac3a5e07 172.16.4.226:6379
   slots:[5249] (1 slots) master
   5 additional replica(s)
S: 43f29b6becef4d5e07d258b0644ff5affee2f6de 172.16.4.222:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
S: d76014a45f6a8d4b64babf07ec6858c1b1e48093 172.16.4.225:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.

4. 出现上图 [ERR] Not all 16384 slots are covered by nodes. 说明slot分配出现问题，可以用fix来依次修复。

[root@redis12 redis]# ./bin/redis-cli --cluster fix 172.16.4.221:6379
# 修复完成后可通过check命令检查修复结果
[root@redis12 redis]# ./bin/redis-cli --cluster check 172.16.4.221:6379
172.16.4.226:6379 (a9790214...) -> 0 keys | 16384 slots | 5 slaves.
[OK] 0 keys in 1 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.16.4.221:6379)
S: ced14cee3bd9d9fe295925e83e18bb4760fd803a 172.16.4.221:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
S: 9ec9ad9d79b50471f1821b9dd4e8be9c90ceac94 172.16.4.223:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
S: 246e69ad920c6c3cf1e152b126deb697b969c611 172.16.4.224:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
M: a9790214d027a76817c89239c003ce42ac3a5e07 172.16.4.226:6379
   slots:[0-16383] (16384 slots) master
   5 additional replica(s)
S: 43f29b6becef4d5e07d258b0644ff5affee2f6de 172.16.4.222:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
S: d76014a45f6a8d4b64babf07ec6858c1b1e48093 172.16.4.225:6379
   slots: (0 slots) slave
   replicates a9790214d027a76817c89239c003ce42ac3a5e07
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看集群

• 客户端可通过cluster info、cluster nodes命令查看集群状态

[root@redis11 redis]# ./bin/redis-cli 
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:1
cluster_current_epoch:6
cluster_my_epoch:6
cluster_stats_messages_ping_sent:1100
cluster_stats_messages_pong_sent:1170
cluster_stats_messages_update_sent:8
cluster_stats_messages_sent:2278
cluster_stats_messages_ping_received:1165
cluster_stats_messages_pong_received:1100
cluster_stats_messages_meet_received:5
cluster_stats_messages_update_received:6
cluster_stats_messages_received:2276
127.0.0.1:6379> cluster nodes
9ec9ad9d79b50471f1821b9dd4e8be9c90ceac94 172.16.4.223:6379@16379 slave a9790214d027a76817c89239c003ce42ac3a5e07 0 1657793873976 6 connected
246e69ad920c6c3cf1e152b126deb697b969c611 172.16.4.224:6379@16379 slave a9790214d027a76817c89239c003ce42ac3a5e07 0 1657793874000 6 connected
a9790214d027a76817c89239c003ce42ac3a5e07 172.16.4.226:6379@16379 master - 0 1657793874987 6 connected 0-16383
43f29b6becef4d5e07d258b0644ff5affee2f6de 172.16.4.222:6379@16379 slave a9790214d027a76817c89239c003ce42ac3a5e07 0 1657793873000 6 connected
ced14cee3bd9d9fe295925e83e18bb4760fd803a 172.16.4.221:6379@16379 myself,slave a9790214d027a76817c89239c003ce42ac3a5e07 0 1657793873000 6 connected
d76014a45f6a8d4b64babf07ec6858c1b1e48093 172.16.4.225:6379@16379 slave a9790214d027a76817c89239c003ce42ac3a5e07 0 1657793872000 6 connected

参考文档

《Redis哨兵+集群模式详解》
《[ERR] Not all 16384 slots are covered by nodes.》