【Redis】Redis-Cluster集群配置搭建

一、背景

Redis常见集群有两种，一种是哨兵，一种是cluster集群
一番搜索得知，哨兵不易扩容，且至少需要双哨兵；所以本例采用官方推荐的cluster模式。

二、环境

Redis版本：5.0.2
5.0以下版本需要安装ruby，本例采用5.0以上版本，不需要单独安装ruby；至少需要5.0以上版本
3台主机6个Redis实例
192.168.1.101:6379
192.168.1.101:6380
192.168.1.102:6379
192.168.1.102:6380
192.168.1.103:6379
192.168.1.103:6380

操作系统redhat7.7

[yyq@localhost redis6379]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)

三、集群配置操作步骤

1、获取redis安装介质

Redis官方下载页面

当前时间官网提供的版本是6.0.9和5.0.10，可以下5.0.10,6.0的版本应该也支持，不过没试过，不完全确定，本例用的是5.0.2
解压

tar -xzvf redis-5.0.2.tar.gz

编译环节参考：

【Redis】Redis安装及常用内容

2、设置目录

创建日志目录、启动命令目录和集群目录,并将redis内容更名复制到两份，分别命名为redis6379和redis6380

[yyq@localhost ]$ pwd
/home/yyq
[yyq@localhost ]$ mkdir logs
[yyq@localhost ]$ mkdir bin
[yyq@localhost ]$ mkdir redis-cluster
[yyq@localhost ]$ mv redis-5.0.2 redis-cluster/
[yyq@localhost ]$ cd redis-cluster
[yyq@localhost ]$ mv redis-5.0.2 redis6379
[yyq@localhost ]$ cp -r redis6379 redis6380

完全配置并启动后目录如下（这是最后的效果）：

[yyq@localhost redis-cluster]$ pwd
/home/yyq/redis-cluster
[yyq@localhost redis-cluster]$ ll
总用量 20
-rw-rw-r-- 1 yyq yyq 92 3月  24 11:35 appendonly.aof
-rw-rw-r-- 1 yyq yyq 175 3月  24 11:35 dump6379.rdb
-rw-r--r-- 1 yyq yyq 175 3月  24 11:35 dump6380.rdb
-rw-r--r-- 1 yyq yyq 805 3月  24 11:36 nodes-6379.conf
-rw-r--r-- 1 yyq yyq 805 3月  24 11:36 nodes-6380.conf
drwxrwxr-x 6 yyq yyq 309 3月  24 14:21 redis6379
drwxrwxr-x 6 yyq yyq 309 3月  24 14:21 redis6380

3、修改配置文件

3.1修改redis.conf

以192.168.1.101为例，102和103除ip地址外，其他保持一致即可

6379端口的文件配置

主要配置

bind 192.168.1.101
port 6379
daemonize yes
pidfile /var/run/redis_6379.pid
logfile "/home/yyq/logs/redis6379.log"
dbfilename dump6379.rdb
dir /home/yyq/redis-cluster
masterauth Yyqabcd543!
requirepass Yyqabcd543!
appendonly yes
appendfilename "appendonly.aof"
cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-node-timeout 15000

6380端口的文件配置

主要配置

bind 192.168.1.101
port 6380
daemonize yes
pidfile /var/run/redis_6380.pid
logfile "/home/yyq/logs/redis6380.log"
dbfilename dump6380.rdb
dir /home/yyq/redis-cluster
masterauth Yyqabcd543!
requirepass Yyqabcd543!
appendonly yes
appendfilename "appendonly.aof"
cluster-enabled yes
cluster-config-file nodes-6380.conf
cluster-node-timeout 15000

详解及说明：

bind 192.168.1.101
cluster-enabled yes #开启cluster
cluster-config-file nodes-6380.conf #自动生成
cluster-node-timeout 15000 #节点通信时间
appendonly yes #持久化方式

比较需要说明的是bind不能写成如下方式绑定多个ip，同一集群内127的重复导致102,103主机的节点一直加不进去
bind 127.0.0. 192.168.1.101

3.2创建启动脚本

/home/yyq/bin目录下创建启动文件 79redis.sh 和 80redis.sh

79redis.sh

/home/yyq/redis-cluster/redis6379/src/redis-server /home/yyq/redis-cluster/redis6379/redis.conf

80redis.sh

/home/yyq/redis-cluster/redis6379/src/redis-server /home/yyq/redis-cluster/redis6379/redis.conf

4.启动redis

分别启动101,102,103三台主机上的6个节点redis
101为例

[yyq@localhost bin]$ ./79redis.sh
[yyq@localhost bin]$ ./80redis.sh

5、开放端口：

每台主机的6379,6380,16379,16380
iptables命令
切换root账号，分别在三台主机开放以下端口

/sbin/iptables -I INPUT -p tcp --dport 6379 -j ACCEPT
/sbin/iptables -I INPUT -p tcp --dport 6380 -j ACCEPT
/sbin/iptables -I INPUT -p tcp --dport 16379 -j ACCEPT
/sbin/iptables -I INPUT -p tcp --dport 16380 -j ACCEPT

redis的每个node启动后占用两个port 6379 & 16379。redis通过port 6379继续对client提供服务，client通过redis独有的文本协议与node进行通信，所以这个port被成为client port or command port。redis node通过port 16379与cluster内部的其他node进行二进制形式的通信，所以被称为data port or bus port。通过port 16379，node之间进行 failure detection(探活)、configure update(配置更新)、failure authorization(失败确认)。如果node使用别的端口作为command port，那么data port 一定是command port + 10000。

6、配置集群

创建集群

101上执行

/home/yyq/redis-cluster/redis6379/src/redis-cli -a Yyqabcd543! --cluster create 192.168.1.101:6379 192.168.1.101:6380 192.168.1.102:6379 192.168.1.102:6380 192.168.1.103:6379 192.168.1.103:6380 --cluster-replicas 1

日志：

[yyq@localhost bin]$ /home/yyq/redis-cluster/redis6379/src/redis-cli -a Yyqabcd543! --cluster create 192.168.1.101:6379 192.168.1.101:6380 192.168.1.102:6379 192.168.1.102:6380 192.168.1.103:6379 192.168.1.103:6380 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.102:6380 to 192.168.1.101:6379
Adding replica 192.168.1.101:6380 to 192.168.1.102:6379
Adding replica 192.168.1.103:6380 to 192.168.1.103:6379
>>> Trying to optimize slaves allocation for anti-affinity
[OK] Perfect anti-affinity obtained!
M: 8862a7713f71e7af471f8bb8de31540f92b1da88 192.168.1.101:6379
   slots:[0-5460] (5461 slots) master
S: a343d174671286d17fc226296d122f61dc34dd09 192.168.1.101:6380
   replicates 2e8734874a26cd9cde1b84713f7298be9f0702d5
M: 503ac7fd38c829609d04d92ca1eaca97b105a078 192.168.1.102:6379
   slots:[5461-10922] (5462 slots) master
S: f989c52edd592ca20b5af5d2841fcf3c51c36ffb 192.168.1.102:6380
   replicates 8862a7713f71e7af471f8bb8de31540f92b1da88
M: 2e8734874a26cd9cde1b84713f7298be9f0702d5 192.168.1.103:6379
   slots:[10923-16383] (5461 slots) master
S: 243ef25caa98ae3b855b5e2ba9aba37bfa6d0247 192.168.1.103:6380
   replicates 503ac7fd38c829609d04d92ca1eaca97b105a078
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.....
>>> Performing Cluster Check (using node 192.168.1.101:6379)
M: 8862a7713f71e7af471f8bb8de31540f92b1da88 192.168.1.101:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: a343d174671286d17fc226296d122f61dc34dd09 192.168.1.101:6380
   slots: (0 slots) slave
   replicates 2e8734874a26cd9cde1b84713f7298be9f0702d5
S: 243ef25caa98ae3b855b5e2ba9aba37bfa6d0247 192.168.1.103:6380
   slots: (0 slots) slave
   replicates 503ac7fd38c829609d04d92ca1eaca97b105a078
M: 2e8734874a26cd9cde1b84713f7298be9f0702d5 192.168.1.103:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 503ac7fd38c829609d04d92ca1eaca97b105a078 192.168.1.102:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: f989c52edd592ca20b5af5d2841fcf3c51c36ffb 192.168.1.102:6380
   slots: (0 slots) slave
   replicates 8862a7713f71e7af471f8bb8de31540f92b1da88
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[ynnms@localhost bin]$ 

验证集群安装情况

任意主机登录其中一节点进行验证

/home/yyq/redis-cluster/redis6379/src/redis-cli -h 192.168.1.101 -p 6380 -a Yyqabcd543!

日志

[yyq@localhost bin]$ /home/yyq/redis-cluster/redis6379/src/redis-cli -h 192.168.1.101 -p 6380 -a Yyqabcd543!
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.101:6380> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:5
cluster_stats_messages_ping_sent:131
cluster_stats_messages_pong_sent:144
cluster_stats_messages_meet_sent:3
cluster_stats_messages_sent:278
cluster_stats_messages_ping_received:142
cluster_stats_messages_pong_received:134
cluster_stats_messages_meet_received:2
cluster_stats_messages_received:278
192.168.1.101:6380> cluster nodes
f989c52edd592ca20b5af5d2841fcf3c51c36ffb 192.168.1.102:6380@16380 slave 8862a7713f71e7af471f8bb8de31540f92b1da88 0 1616557083876 4 connected
243ef25caa98ae3b855b5e2ba9aba37bfa6d0247 192.168.1.103:6380@16380 slave 503ac7fd38c829609d04d92ca1eaca97b105a078 0 1616557081000 6 connected
2e8734874a26cd9cde1b84713f7298be9f0702d5 192.168.1.103:6379@16379 master - 0 1616557084879 5 connected 10923-16383
503ac7fd38c829609d04d92ca1eaca97b105a078 192.168.1.102:6379@16379 master - 0 1616557082872 3 connected 5461-10922
a343d174671286d17fc226296d122f61dc34dd09 192.168.1.101:6380@16380 myself,slave 2e8734874a26cd9cde1b84713f7298be9f0702d5 0 1616557083000 2 connected
8862a7713f71e7af471f8bb8de31540f92b1da88 192.168.1.101:6379@16379 master - 0 1616557082000 1 connected 0-5460
192.168.1.101:6380> quit
[yyq@localhost bin]$ 

完成

参考：三个节点创建集群日志

1节点日志

==> redis6379.log <==
21210:M 24 Mar 2021 11:35:53.054 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH

==> redis6380.log <==
21216:M 24 Mar 2021 11:35:53.054 # configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH

==> redis6379.log <==
21210:M 24 Mar 2021 11:35:53.066 # IP address for this node updated to 192.168.1.101

==> redis6380.log <==
21216:M 24 Mar 2021 11:35:53.196 # IP address for this node updated to 192.168.1.101

==> redis6379.log <==
21210:M 24 Mar 2021 11:35:58.015 # Cluster state changed: ok

==> redis6380.log <==
21216:M 24 Mar 2021 11:35:58.153 # Cluster state changed: ok
21216:S 24 Mar 2021 11:35:59.069 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
21216:S 24 Mar 2021 11:35:59.557 * Connecting to MASTER 192.168.1.103:6379
21216:S 24 Mar 2021 11:35:59.558 * MASTER <-> REPLICA sync started
21216:S 24 Mar 2021 11:35:59.558 * Non blocking connect for SYNC fired the event.
21216:S 24 Mar 2021 11:35:59.559 * Master replied to PING, replication can continue...
21216:S 24 Mar 2021 11:35:59.559 * Trying a partial resynchronization (request 83144a1c64df96ea8fb67e25a3bb010b7a732d09:1).
21216:S 24 Mar 2021 11:35:59.561 * Full resync from master: dc20b1b30000e221cfa6ef2ec389835f81a336b7:0
21216:S 24 Mar 2021 11:35:59.561 * Discarding previously cached master state.
21216:S 24 Mar 2021 11:35:59.605 * MASTER <-> REPLICA sync: receiving 175 bytes from master
21216:S 24 Mar 2021 11:35:59.605 * MASTER <-> REPLICA sync: Flushing old data
21216:S 24 Mar 2021 11:35:59.605 * MASTER <-> REPLICA sync: Loading DB in memory
21216:S 24 Mar 2021 11:35:59.605 * MASTER <-> REPLICA sync: Finished with success
21216:S 24 Mar 2021 11:35:59.606 * Background append only file rewriting started by pid 21241
21216:S 24 Mar 2021 11:35:59.638 * AOF rewrite child asks to stop sending diffs.
21241:C 24 Mar 2021 11:35:59.638 * Parent agreed to stop sending diffs. Finalizing AOF...
21241:C 24 Mar 2021 11:35:59.638 * Concatenating 0.00 MB of AOF diff received from parent.
21241:C 24 Mar 2021 11:35:59.638 * SYNC append only file rewrite performed
21241:C 24 Mar 2021 11:35:59.639 * AOF rewrite: 4 MB of memory used by copy-on-write
21216:S 24 Mar 2021 11:35:59.658 * Background AOF rewrite terminated with success
21216:S 24 Mar 2021 11:35:59.658 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
21216:S 24 Mar 2021 11:35:59.658 * Background AOF rewrite finished successfully

==> redis6379.log <==
21210:M 24 Mar 2021 11:35:59.780 * Replica 192.168.1.102:6380 asks for synchronization
21210:M 24 Mar 2021 11:35:59.780 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '72e72ec5ef694aaf698f6a1d75a36737ab0ea571', my replication IDs are '79abe38366dc1e174dea1117920a5daf88de9b9e' and '0000000000000000000000000000000000000000')
21210:M 24 Mar 2021 11:35:59.780 * Starting BGSAVE for SYNC with target: disk
21210:M 24 Mar 2021 11:35:59.781 * Background saving started by pid 21242
21242:C 24 Mar 2021 11:35:59.783 * DB saved on disk
21242:C 24 Mar 2021 11:35:59.784 * RDB: 4 MB of memory used by copy-on-write
21210:M 24 Mar 2021 11:35:59.821 * Background saving terminated with success
21210:M 24 Mar 2021 11:35:59.822 * Synchronization with replica 192.168.1.102:6380 succeeded

2节点日志

==> redis6379.log <==
21722:M 24 Mar 2021 11:35:53.054 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH

==> redis6380.log <==
21729:M 24 Mar 2021 11:35:53.055 # configEpoch set to 4 via CLUSTER SET-CONFIG-EPOCH

==> redis6379.log <==
21722:M 24 Mar 2021 11:35:53.096 # IP address for this node updated to 192.168.1.102

==> redis6380.log <==
21729:M 24 Mar 2021 11:35:53.196 # IP address for this node updated to 192.168.1.102
21729:M 24 Mar 2021 11:35:58.172 # Cluster state changed: ok

==> redis6379.log <==
21722:M 24 Mar 2021 11:35:58.688 # Cluster state changed: ok

==> redis6380.log <==
21729:S 24 Mar 2021 11:35:59.070 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.

==> redis6379.log <==
21722:M 24 Mar 2021 11:35:59.612 * Replica 192.168.1.103:6380 asks for synchronization
21722:M 24 Mar 2021 11:35:59.612 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '492d5fe07197779c7e7a75f3f88466ba06c1b44d', my replication IDs are '172b5439c9d43592488fd610244e535d2deafbe7' and '0000000000000000000000000000000000000000')
21722:M 24 Mar 2021 11:35:59.612 * Starting BGSAVE for SYNC with target: disk
21722:M 24 Mar 2021 11:35:59.617 * Background saving started by pid 21764
21764:C 24 Mar 2021 11:35:59.621 * DB saved on disk
21764:C 24 Mar 2021 11:35:59.622 * RDB: 4 MB of memory used by copy-on-write
21722:M 24 Mar 2021 11:35:59.690 * Background saving terminated with success
21722:M 24 Mar 2021 11:35:59.691 * Synchronization with replica 192.168.1.103:6380 succeeded

==> redis6380.log <==
21729:S 24 Mar 2021 11:35:59.777 * Connecting to MASTER 192.168.1.101:6379
21729:S 24 Mar 2021 11:35:59.777 * MASTER <-> REPLICA sync started
21729:S 24 Mar 2021 11:35:59.778 * Non blocking connect for SYNC fired the event.
21729:S 24 Mar 2021 11:35:59.778 * Master replied to PING, replication can continue...
21729:S 24 Mar 2021 11:35:59.779 * Trying a partial resynchronization (request 72e72ec5ef694aaf698f6a1d75a36737ab0ea571:1).
21729:S 24 Mar 2021 11:35:59.782 * Full resync from master: 29b2cc29b7a0aa91c9ef8c0278c4c6b8986536e4:0
21729:S 24 Mar 2021 11:35:59.782 * Discarding previously cached master state.
21729:S 24 Mar 2021 11:35:59.821 * MASTER <-> REPLICA sync: receiving 175 bytes from master
21729:S 24 Mar 2021 11:35:59.822 * MASTER <-> REPLICA sync: Flushing old data
21729:S 24 Mar 2021 11:35:59.822 * MASTER <-> REPLICA sync: Loading DB in memory
21729:S 24 Mar 2021 11:35:59.822 * MASTER <-> REPLICA sync: Finished with success
21729:S 24 Mar 2021 11:35:59.823 * Background append only file rewriting started by pid 21765
21729:S 24 Mar 2021 11:35:59.851 * AOF rewrite child asks to stop sending diffs.
21765:C 24 Mar 2021 11:35:59.852 * Parent agreed to stop sending diffs. Finalizing AOF...
21765:C 24 Mar 2021 11:35:59.852 * Concatenating 0.00 MB of AOF diff received from parent.
21765:C 24 Mar 2021 11:35:59.852 * SYNC append only file rewrite performed
21765:C 24 Mar 2021 11:35:59.852 * AOF rewrite: 4 MB of memory used by copy-on-write
21729:S 24 Mar 2021 11:35:59.878 * Background AOF rewrite terminated with success
21729:S 24 Mar 2021 11:35:59.878 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
21729:S 24 Mar 2021 11:35:59.878 * Background AOF rewrite finished successfully

3节点日志

==> redis6379.log <==
84658:M 24 Mar 2021 11:35:53.056 # configEpoch set to 5 via CLUSTER SET-CONFIG-EPOCH

==> redis6380.log <==
84664:M 24 Mar 2021 11:35:53.056 # configEpoch set to 6 via CLUSTER SET-CONFIG-EPOCH

==> redis6379.log <==
84658:M 24 Mar 2021 11:35:53.096 # IP address for this node updated to 10.173.14.143

==> redis6380.log <==
84664:M 24 Mar 2021 11:35:53.096 # IP address for this node updated to 10.173.14.143

==> redis6379.log <==
84658:M 24 Mar 2021 11:35:58.000 # Cluster state changed: ok

==> redis6380.log <==
84664:M 24 Mar 2021 11:35:58.506 # Cluster state changed: ok
84664:S 24 Mar 2021 11:35:59.071 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.

==> redis6379.log <==
84658:M 24 Mar 2021 11:35:59.559 * Replica 10.173.14.141:6380 asks for synchronization
84658:M 24 Mar 2021 11:35:59.560 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '83144a1c64df96ea8fb67e25a3bb010b7a732d09', my replication IDs are '1ef84529c2b638a1d544209c7cb8149137646642' and '0000000000000000000000000000000000000000')
84658:M 24 Mar 2021 11:35:59.560 * Starting BGSAVE for SYNC with target: disk
84658:M 24 Mar 2021 11:35:59.561 * Background saving started by pid 84708
84708:C 24 Mar 2021 11:35:59.562 * DB saved on disk
84708:C 24 Mar 2021 11:35:59.563 * RDB: 2 MB of memory used by copy-on-write
84658:M 24 Mar 2021 11:35:59.604 * Background saving terminated with success
84658:M 24 Mar 2021 11:35:59.604 * Synchronization with replica 10.173.14.141:6380 succeeded

==> redis6380.log <==
84664:S 24 Mar 2021 11:35:59.610 * Connecting to MASTER 10.173.14.142:6379
84664:S 24 Mar 2021 11:35:59.610 * MASTER <-> REPLICA sync started
84664:S 24 Mar 2021 11:35:59.611 * Non blocking connect for SYNC fired the event.
84664:S 24 Mar 2021 11:35:59.611 * Master replied to PING, replication can continue...
84664:S 24 Mar 2021 11:35:59.612 * Trying a partial resynchronization (request 492d5fe07197779c7e7a75f3f88466ba06c1b44d:1).
84664:S 24 Mar 2021 11:35:59.619 * Full resync from master: b008bbf1c5ad052d910cab2e26532293c8c9d57c:0
84664:S 24 Mar 2021 11:35:59.619 * Discarding previously cached master state.
84664:S 24 Mar 2021 11:35:59.691 * MASTER <-> REPLICA sync: receiving 175 bytes from master
84664:S 24 Mar 2021 11:35:59.692 * MASTER <-> REPLICA sync: Flushing old data
84664:S 24 Mar 2021 11:35:59.692 * MASTER <-> REPLICA sync: Loading DB in memory
84664:S 24 Mar 2021 11:35:59.696 * MASTER <-> REPLICA sync: Finished with success
84664:S 24 Mar 2021 11:35:59.702 * Background append only file rewriting started by pid 84709
84664:S 24 Mar 2021 11:35:59.730 * AOF rewrite child asks to stop sending diffs.
84709:C 24 Mar 2021 11:35:59.731 * Parent agreed to stop sending diffs. Finalizing AOF...
84709:C 24 Mar 2021 11:35:59.731 * Concatenating 0.00 MB of AOF diff received from parent.
84709:C 24 Mar 2021 11:35:59.731 * SYNC append only file rewrite performed
84709:C 24 Mar 2021 11:35:59.732 * AOF rewrite: 6 MB of memory used by copy-on-write
84664:S 24 Mar 2021 11:35:59.815 * Background AOF rewrite terminated with success
84664:S 24 Mar 2021 11:35:59.815 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
84664:S 24 Mar 2021 11:35:59.815 * Background AOF rewrite finished successfully

四、后续添加节点

后续可通郭以下命令添加节点

CLUSTER MEET 192.168.1.104 6379