redis 集群简介

备注:测试版本 redis 4.0.9

一.Cluster模式介绍

sentinel模式基本可以满足一般生产的需求，具备高可用性。但是当数据量过大到一台服务器存放不下的情况时，主从模式或sentinel模式就不能满足需求了，这个时候需要对存储的数据进行分片，将数据存储到多个Redis实例中。cluster模式的出现就是为了解决单机Redis容量有限的问题，将Redis的数据根据一定的规则分配到多台机器。

cluster可以说是sentinel和主从模式的结合体，通过cluster可以实现主从和master重选功能，所以如果配置两个副本三个分片的话，就需要六个Redis实例。因为Redis的数据是根据一定规则分配到cluster的不同机器的，当数据量过大时，可以新增机器进行扩容。

使用集群，只需要将redis配置文件中的cluster-enable配置打开即可。每个集群中至少需要三个主数据库才能正常运行，新增节点非常方便。

cluster集群特点：
1.多个redis节点网络互联，数据共享
2.所有的节点都是一主一从（也可以是一主多从），其中从不提供服务，仅作为备用
3.不支持同时处理多个key（如MSET/MGET），因为redis需要把key均匀分布在各个节点上，
4.并发量很高的情况下同时创建key-value会降低性能并导致不可预测的行为
5.支持在线增加、删除节点
6.客户端可以连接任何一个主节点进行读写

二.redis三主三从集群搭建

服务器类别	IP	端口
主	10.31.1.123	7000
从	10.31.1.123	7001
主	10.31.1.124	7002
从	10.31.1.124	7003
主	10.31.1.125	7004
从	10.31.1.125	7005

服务器及端口配置如上，此时 123、124、125三台服务器已经安装redis

2.1 配置文件

创建目录

-- 123
mkdir -p /usr/local/redis/7000/conf
mkdir -p /usr/local/redis/7000/log
mkdir -p /usr/local/redis/7001/conf
mkdir -p /usr/local/redis/7001/log

-- 124 
mkdir -p /usr/local/redis/7002/conf
mkdir -p /usr/local/redis/7002/log
mkdir -p /usr/local/redis/7003/conf
mkdir -p /usr/local/redis/7003/log

-- 124
mkdir -p /usr/local/redis/7004/conf
mkdir -p /usr/local/redis/7004/log
mkdir -p /usr/local/redis/7005/conf
mkdir -p /usr/local/redis/7005/log

10.31.1.123上主库端口7000的配置文件：
protected-mode no 、## requirepass 123456 这两个我测试过程中，会影响到后面的集群初始化，如果需要提高安全等级，可以在配置完集群之后，再开启

## SHOW REDIS LOGO ##
always-show-logo yes

## LOCAL CONNECT
bind 10.31.1.123
protected-mode no

## GENERAL ##
daemonize yes
supervised no
pidfile /usr/local/redis/7000/redis.pid
port 7000
timeout 0
tcp-keepalive 300
loglevel notice
logfile "/usr/local/redis/7000/log/redis.log"
databases 16

## SNAPSHOTTING ##
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /usr/local/redis/7000/

## REPLICATION ##
#slaveof MASTER_IP 6380
masterauth 123456
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100

## SECURITY ##
## requirepass 123456

## LIMITS ##
#maxclients 10000
#maxmemory <bytes>

## APPEND ONLY MODE ##
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble no

## LUA SCRIPTING ##
lua-time-limit 5000

## SLOW LOG ##
slowlog-log-slower-than 10000
slowlog-max-len 128

## EVENT NOTIFICATION ##
notify-keyspace-events ""

## ADVANCED CONFIG ##
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

## EXPIRED DATA AND ELIMINATION STRATEGY ##
slave-lazy-flush no
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no

## DELAY MONITORING ##
latency-monitor-threshold 0

## cluster ##
cluster-enabled yes
cluster-config-file /usr/local/redis/7000/conf/nodes.conf
cluster-node-timeout 5000

10.31.1.123上从库端口7001的配置文件：
区别两点:
1.端口号从7000改为7001

## SHOW REDIS LOGO ##
always-show-logo yes

## LOCAL CONNECT
bind 10.31.1.123
protected-mode no

## GENERAL ##
daemonize yes
supervised no
pidfile /usr/local/redis/7001/redis.pid
port 7001
timeout 0
tcp-keepalive 300
loglevel notice
logfile "/usr/local/redis/7001/log/redis.log"
databases 16

## SNAPSHOTTING ##
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /usr/local/redis/7001/

## REPLICATION ##
#slaveof MASTER_IP 6380
masterauth 123456
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100

## SECURITY ##
## requirepass 123456

## LIMITS ##
#maxclients 10000
#maxmemory <bytes>

## APPEND ONLY MODE ##
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble no

## LUA SCRIPTING ##
lua-time-limit 5000

## SLOW LOG ##
slowlog-log-slower-than 10000
slowlog-max-len 128

## EVENT NOTIFICATION ##
notify-keyspace-events ""

## ADVANCED CONFIG ##
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

## EXPIRED DATA AND ELIMINATION STRATEGY ##
slave-lazy-flush no
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no

## DELAY MONITORING ##
latency-monitor-threshold 0

## cluster ##
cluster-enabled yes
cluster-config-file /usr/local/redis/7001/conf/nodes.conf
cluster-node-timeout 5000

10.31.1.124、10.31.1.125两台上的配置文件同10.31.1.123，只需要改对应的端口号即可

2.2 启动redis服务

-- 123
nohup redis-server /usr/local/redis/7000/conf/7000.conf &
nohup redis-server /usr/local/redis/7001/conf/7001.conf &
-- 124
nohup redis-server /usr/local/redis/7002/conf/7002.conf &
nohup redis-server /usr/local/redis/7003/conf/7003.conf &
-- 125
nohup redis-server /usr/local/redis/7004/conf/7004.conf &
nohup redis-server /usr/local/redis/7005/conf/7005.conf &

查看redis服务是否启动
如下所示，则表示均已经启动成功了

[root@10-31-1-123 conf]# ps -ef | grep redis
root     22166     1  0 16:21 ?        00:00:00 redis-server 10.31.1.123:7000 [cluster]
root     22318     1  0 16:24 ?        00:00:00 redis-server 10.31.1.123:7001 [cluster]
root     22329 18004  0 16:24 pts/0    00:00:00 grep --color=auto redis

[root@10-31-1-124 conf]# ps -ef | grep redis
root      9971     1  0 16:22 ?        00:00:00 redis-server 10.31.1.124:7002 [cluster]
root     10124     1  0 16:25 ?        00:00:00 redis-server 10.31.1.124:7003 [cluster]
root     10132  8437  0 16:25 pts/0    00:00:00 grep --color=auto redis

[root@10-31-1-125 conf]# ps -ef | grep redis
root      5725     1  0 16:22 ?        00:00:00 redis-server 10.31.1.125:7004 [cluster]
root      5740     1  0 16:26 ?        00:00:00 redis-server 10.31.1.125:7005 [cluster]
root      5745  5651  0 16:26 pts/0    00:00:00 grep --color=auto redis

2.3 安装ruby

10.31.1.123上执行即可

yum -y install zlib ruby rubygems
gem install redis

报错:

[root@10-31-1-123 ~]# gem install redis
Fetching: redis-4.2.2.gem (100%)
ERROR:  Error installing redis:
        redis requires Ruby version >= 2.3.0.
[root@10-31-1-123 ~]# 

参考如下解决方案:

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
curl -sSL https://get.rvm.io | bash -s stable
find / -name rvm -print
source /usr/local/rvm/scripts/rvm
rvm list known
rvm install 2.5.1
rvm use 2.5.1
rvm use 2.5.1 --default
rvm remove 2.0.0
ruby --version
gem install redis

2.4 初始化集群

/usr/local/src/redis-4.0.9/src/redis-trib.rb create --replicas 1 10.31.1.123:7000 10.31.1.123:7001 10.31.1.124:7002 10.31.1.124:7003 10.31.1.125:7004 10.31.1.125:7005

测试记录:

[root@10-31-1-123 local]# /usr/local/src/redis-4.0.9/src/redis-trib.rb create --replicas 1 10.31.1.123:7000 10.31.1.123:7001 10.31.1.124:7002 10.31.1.124:7003 10.31.1.125:7004 10.31.1.125:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.31.1.123:7000
10.31.1.124:7002
10.31.1.125:7004
Adding replica 10.31.1.124:7003 to 10.31.1.123:7000
Adding replica 10.31.1.125:7005 to 10.31.1.124:7002
Adding replica 10.31.1.123:7001 to 10.31.1.125:7004
M: 247b6072aa240ba66703403466c159158dd880a4 10.31.1.123:7000
   slots:0-5460 (5461 slots) master
S: 47257b5838640c6736cf3b5eb6eaf87d0f11fece 10.31.1.123:7001
   replicates df1747df706f01a6a671f390af9f56659874b959
M: c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea 10.31.1.124:7002
   slots:5461-10922 (5462 slots) master
S: e7f72bdf6efb013017c3942855507bffac4ed979 10.31.1.124:7003
   replicates 247b6072aa240ba66703403466c159158dd880a4
M: df1747df706f01a6a671f390af9f56659874b959 10.31.1.125:7004
   slots:10923-16383 (5461 slots) master
S: 14e1ff83ef179a2a5661b11de4e0412993eb466b 10.31.1.125:7005
   replicates c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 10.31.1.123:7000)
M: 247b6072aa240ba66703403466c159158dd880a4 10.31.1.123:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: e7f72bdf6efb013017c3942855507bffac4ed979 10.31.1.124:7003
   slots: (0 slots) slave
   replicates 247b6072aa240ba66703403466c159158dd880a4
S: 14e1ff83ef179a2a5661b11de4e0412993eb466b 10.31.1.125:7005
   slots: (0 slots) slave
   replicates c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea
M: df1747df706f01a6a671f390af9f56659874b959 10.31.1.125:7004
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea 10.31.1.124:7002
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 47257b5838640c6736cf3b5eb6eaf87d0f11fece 10.31.1.123:7001
   slots: (0 slots) slave
   replicates df1747df706f01a6a671f390af9f56659874b959
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@10-31-1-123 local]# 

下面根据输出信息详细介绍集群创建的过程。

首先redis-trib.rb会以客户端的形式尝试连接所有的节点，并发送ping命令以确定节点能够正常服务。如果有任何节点无法连接，则创建失败。同时发送info命令获取每个节点的运行ID以及是否开启了集群功能（即cluster_enabled为1）。

准备就绪后集群会向每个节点发送cluster meet命令，格式为cluster meet ip port，这个命令用来告诉当前节点指定ip和port上在运行的节点也是集群的一部分，从而使得6个节点最终可以归入一个集群。

然后redis-trib.rb会分配主从库节点，分配的原则是尽量保证每个主库运行在不同的IP地址上，同时每个从库和主库均不运行在同一IP地址上，以保证系统的容灾能力。分配结果如下：

Using 3 masters:
10.31.1.123:7000
10.31.1.124:7002
10.31.1.125:7004
Adding replica 10.31.1.124:7003 to 10.31.1.123:7000
Adding replica 10.31.1.125:7005 to 10.31.1.124:7002
Adding replica 10.31.1.123:7001 to 10.31.1.125:7004

主从分配完成后，会为每个主库分配插槽，分配插槽的过程其实就是分配哪些键归哪些节点负责。之后对每个要成为从库的节点发送cluster replicate 主库运行ID，来将当前节点转换成从库并复制指定运行ID的主库节点。

此时整个集群的过程即创建完成，使用redis命令行客户端连接任意一个节点执行cluster nodes可以获得集群中的所有节点信息，如在10.31.1.123执行：

[root@10-31-1-123 ~]# redis-cli -h 10.31.1.123 -p 7000 cluster nodes
e7f72bdf6efb013017c3942855507bffac4ed979 10.31.1.124:7003@17003 slave 247b6072aa240ba66703403466c159158dd880a4 0 1604483299146 4 connected
14e1ff83ef179a2a5661b11de4e0412993eb466b 10.31.1.125:7005@17005 slave c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea 0 1604483300149 6 connected
df1747df706f01a6a671f390af9f56659874b959 10.31.1.125:7004@17004 master - 0 1604483300550 5 connected 10923-16383
c6e86ccf2b3d5002f1640c2ad8ef397dead1f4ea 10.31.1.124:7002@17002 master - 0 1604483299000 3 connected 5461-10922
247b6072aa240ba66703403466c159158dd880a4 10.31.1.123:7000@17000 myself,master - 0 1604483300000 1 connected 0-5460
47257b5838640c6736cf3b5eb6eaf87d0f11fece 10.31.1.123:7001@17001 slave df1747df706f01a6a671f390af9f56659874b959 0 1604483301150 5 connected
[root@10-31-1-123 ~]# 

参考

1.https://blog.csdn.net/miss1181248983/article/details/90056960
2.https://blog.csdn.net/wzy0623/article/details/82706614