Redis学习笔记（五）—— 在Linux下搭建Redis集群

搭建Redis单机版请参考： Redis学习笔记（二）— 在linux下搭建redis服务器

一、Redis集群的工作原理

1、所有的redis节点彼此之间通过ping-pong机制互联，内部使用二进制协议优化传输速度和带宽；

2、redis客户端与redis节点直接连接，不需要中间代理层，redis客户端只要连接上redis集群中的任何一个节点即可使用，不需要连接所有节点；

3、redis集群中内置了16384个哈希槽，当需要在redis集群中放置一个key-value时，redis先对key使用crc16算法算出一个结果，然后把结果对16384求余数，这样每一个key都会对应一个编号在0-16383之间的哈希槽，redis会根据节点的数量大致均等的将哈希槽映射到不同的节点。

二、Redis-cluster的投票机制

1、投票过程是集群中的所有节点全部参与，如果半数以上的节点与另一节点通信超时，则认为这个节点挂掉；

2、什么时候整个集群进入fail状态？

① 如果集群任意master（主节点）挂掉，并且当前master没有slave（从节点），集群进入fail状态，也可以理解为集群的哈希槽映射（0-16383）不完整时进入fail状态；

② 如果集群中超过半数以上的master挂掉，无论是否有slave，集群进入fail状态；

当集群进入fail状态时，所有对集群的操作都不可用，会收到error提示。

三、搭建Redis集群

1、下载ruby（ruby是一个面向对象程序设计的脚本语言）

[root@localhost ~]# yum install ruby

2、下载ruby与redis的接口程序并上传到root目录

下载地址：https://rubygems.org/gems/redis/versions/4.0.0

3、安装ruby与redis的接口程序

[root@localhost ~]# gem install redis-4.0.0.gem
Successfully installed redis-4.0.0
Parsing documentation for redis-4.0.0
Installing ri documentation for redis-4.0.0
Done installing documentation for redis after 1 seconds
1 gem installed

如果遇到error：redis requires Ruby version >= 2.2.2.

请参考：Redis学习笔记（六）—— 安装ruby出现问题：redis requires Ruby version >= 2.2.2.

4、在/usr/local目录下创建redis-cluster目录

[root@localhost ~]# mkdir /usr/local/redis-cluster

5、将单机版redis目录下的bin目录复制到redis-cluster目录下，重命名为redis1

[root@localhost ~]# cp -r /usr/local/redis/bin /usr/local/redis-cluster/redis1

6、然后在redis-cluster目录下再复制5个redis目录，一共是6个redis实例，目录结构如下

[root@localhost ~]# ls /usr/local/redis-cluster
redis1  redis2  redis3  redis4  redis5  redis6

7、分别修改6个redis实例下面的配置文件redis.conf

[root@localhost ~]# vi /usr/local/redis-cluster/redis1/redis.conf

① 修改端口号（分别修改为8001-8006）

② 去掉cluster-enabled yes前面的注释，代表开启集群服务

8、进入到解压的redis源码src目录下将redis-trib.rb脚本复制到redis-cluster目录下

[root@localhost ~]# cd redis-4.0.6/src
[root@localhost src]# cp redis-trib.rb /usr/local/redis-cluster

9、将redis1-6目录中的dump.rdb文件删掉（防止搭建集群出现error，没有的不管）

[root@localhost redis-cluster]# rm -rf redis1/dump.rdb

10、在redis-cluster目录下创建一个快速启动脚本

[root@localhost redis-cluster]# vi startall.sh

for a in 1 2 3 4 5 6 
do
cd redis$a
./redis-server redis.conf
cd ..
done

11、赋予脚本执行权限

[root@localhost redis-cluster]# chmod +x startall.sh

12、启动脚本

[root@localhost redis-cluster]# ./startall.sh

13、查看进程（6个redis实例均已启动）

[root@localhost redis-cluster]# ps aux|grep redis
root       1486  0.0  0.9 151404  9692 ?        Ssl  10:32   0:00 ./redis-server *:8001 [cluster]
root       1491  0.0  0.9 147308  9680 ?        Ssl  10:32   0:00 ./redis-server *:8002 [cluster]
root       1493  0.0  0.9 147308  9676 ?        Ssl  10:32   0:00 ./redis-server *:8003 [cluster]
root       1498  0.0  0.9 147308  9676 ?        Ssl  10:32   0:00 ./redis-server *:8004 [cluster]
root       1503  0.0  0.9 147308  9680 ?        Ssl  10:32   0:00 ./redis-server *:8005 [cluster]
root       1508  0.0  0.9 147308  9680 ?        Ssl  10:32   0:00 ./redis-server *:8006 [cluster]

14、执行ruby脚本，创建集群

[root@localhost redis-cluster]# ./redis-trib.rb create --replicas 1 192.168.188.130:8001 192.168.188.130:8002 192.168.188.130:8003 192.168.188.130:8004 192.168.188.130:8005  192.168.188.130:8006
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.188.130:8001
192.168.188.130:8002
192.168.188.130:8003
Adding replica 192.168.188.130:8004 to 192.168.188.130:8001
Adding replica 192.168.188.130:8005 to 192.168.188.130:8002
Adding replica 192.168.188.130:8006 to 192.168.188.130:8003

输入yes

说明：redis集群中至少需要3个节点，每个主节点都有一个从节点，一共是6个节点，replicas为1是指定每个主节点有一个从节点

创建集群如果出现如下错误：

[ERR] Node 192.168.188.130:8001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

可以尝试删除每个redis目录下的nodes.conf、dump.rdb文件，然后重新启动redis实例，如果还不行的话就删掉所有的redis目录，重新按步骤再来一次

15、连接测试

[root@localhost redis-cluster]# redis1/redis-cli -h 192.168.188.130 -p 8005 -c
192.168.188.130:8005> set a 10
-> Redirected to slot [15495] located at 192.168.188.130:8003
OK
192.168.188.130:8003> get a
"10"

-h 代表主机，-p 代表端口号，-c 代表连接

16、查看集群节点（cluster info可以查看节点状态）

192.168.188.130:8003> cluster nodes
c2dae95621795fa6cde2e740b2aa5eab06257b38 192.168.188.130:8005@18005 slave e115b7ba180a06939ef1dea97fbcd0e6d5afc375 0 1520612818000 5 connected
e115b7ba180a06939ef1dea97fbcd0e6d5afc375 192.168.188.130:8002@18002 master - 0 1520612819000 2 connected 5461-10922 //哈希槽
86109bfab2cffae0a2f3daf9e8a8a39c7b8e0277 192.168.188.130:8003@18003 myself,master - 0 1520612818000 3 connected 10923-16383 //哈希槽
6d064a0b40f0cee1c6f067012998bd74b4422f79 192.168.188.130:8004@18004 slave ae84008e643f25c7547071b7068882d16769d11c 0 1520612817435 4 connected
f4968d5a47fb7cefb7d4b9422db3de1d73f3e8d6 192.168.188.130:8006@18006 slave 86109bfab2cffae0a2f3daf9e8a8a39c7b8e0277 0 1520612819450 6 connected
ae84008e643f25c7547071b7068882d16769d11c 192.168.188.130:8001@18001 master - 0 1520612816000 1 connected 0-5460 //哈希槽

17、添加新的主节点

按照之前的步骤，复制一个redis7目录，修改redis7目录下的redis.conf文件，启动redis7实例，然后执行如下命令：

[root@localhost redis-cluster]# ./redis-trib.rb add-node 192.168.188.130:8007 192.168.188.130:8001
>>> Adding node 192.168.188.130:8007 to cluster 192.168.188.130:8001
>>> Performing Cluster Check (using node 192.168.188.130:8001)

节点创建成功，但这时候还不能存储数据，还需要分配哈希槽

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.188.130:8007 to make it join the cluster.
[OK] New node added correctly.

18、给主节点分配哈希槽

① 使用ruby命令连接上哈希槽（连接任何一个节点都可以）

[root@localhost redis-cluster]# ./redis-trib.rb reshard 192.168.188.130:8002                     
>>> Performing Cluster Check (using node 192.168.188.130:8002)
M: e115b7ba180a06939ef1dea97fbcd0e6d5afc375 192.168.188.130:8002
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 6d064a0b40f0cee1c6f067012998bd74b4422f79 192.168.188.130:8004
   slots: (0 slots) slave
   replicates ae84008e643f25c7547071b7068882d16769d11c
M: 15a6627bee030631dd7862c320174e479fc644a8 192.168.188.130:8007
   slots: (0 slots) master
   0 additional replica(s)
S: f4968d5a47fb7cefb7d4b9422db3de1d73f3e8d6 192.168.188.130:8006
   slots: (0 slots) slave
   replicates 86109bfab2cffae0a2f3daf9e8a8a39c7b8e0277
M: 86109bfab2cffae0a2f3daf9e8a8a39c7b8e0277 192.168.188.130:8003
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: ae84008e643f25c7547071b7068882d16769d11c 192.168.188.130:8001
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: c2dae95621795fa6cde2e740b2aa5eab06257b38 192.168.188.130:8005
   slots: (0 slots) slave
   replicates e115b7ba180a06939ef1dea97fbcd0e6d5afc375
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 

② 输入要分配的槽数量（输入1000即代表1000个槽）

How many slots do you want to move (from 1 to 16384)? 1000

③ 输入准备接收槽的节点id（输入8007前面的id编号）

What is the receiving node ID? 15a6627bee030631dd7862c320174e479fc644a8

④ 输入源节点的id，槽将从源节点拿，分配后槽在源节点中就不存在了，all代表从所有源节点中拿，done取消

Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:all

⑤ 输入yes开始移动槽到目标节点

Do you want to proceed with the proposed reshard plan (yes/no)? yes

⑥ 查看集群节点（8007中有1000个槽可用了）

[root@localhost redis-cluster]# redis1/redis-cli -h 192.168.188.130 -p 8001 -c 
192.168.188.130:8001> cluster nodes
f4968d5a47fb7cefb7d4b9422db3de1d73f3e8d6 192.168.188.130:8006@18006 slave 86109bfab2cffae0a2f3daf9e8a8a39c7b8e0277 0 1520616291692 6 connected
c2dae95621795fa6cde2e740b2aa5eab06257b38 192.168.188.130:8005@18005 slave e115b7ba180a06939ef1dea97fbcd0e6d5afc375 0 1520616288671 5 connected
ae84008e643f25c7547071b7068882d16769d11c 192.168.188.130:8001@18001 myself,master - 0 1520616290000 1 connected 333-5460
e115b7ba180a06939ef1dea97fbcd0e6d5afc375 192.168.188.130:8002@18002 master - 0 1520616290000 2 connected 5795-10922
15a6627bee030631dd7862c320174e479fc644a8 192.168.188.130:8007@18007 master - 0 1520616290684 7 connected 0-332 5461-5794 10923-11255
86109bfab2cffae0a2f3daf9e8a8a39c7b8e0277 192.168.188.130:8003@18003 master - 0 1520616289000 3 connected 11256-16383
6d064a0b40f0cee1c6f067012998bd74b4422f79 192.168.188.130:8004@18004 slave ae84008e643f25c7547071b7068882d16769d11c 0 1520616289000 4 connected

19、添加从节点

按照上面的步骤，再创建一个实例redis8，启动后执行如下命令（后面三个参数分别是主节点id编号，要添加的从节点ip端口，集群中已有的节点ip端口）

[root@localhost redis-cluster]# ./redis-trib.rb add-node --slave --master-id 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483 192.168.188.130:8008 192.168.188.130:8002
>>> Adding node 192.168.188.130:8008 to cluster 192.168.188.130:8002
>>> Performing Cluster Check (using node 192.168.188.130:8002)

从节点添加成功

[root@localhost redis-cluster]# redis1/redis-cli -h 192.168.188.130 -p 8005 -c
192.168.188.130:8005> cluster nodes
eeda69f0090ce204714d0b96d1e5565349a97a8a 192.168.188.130:8006@18006 slave d2dd4adcadf9a76bcee03dd2ed2881cc980b93c1 0 1520618414083 6 connected
377cb5f5e72e12d6c7cc6d90f3c3768f66f50483 192.168.188.130:8007@18007 master - 0 1520618413076 7 connected 0-332 5461-5794 10923-11255
719191fa7797cde14ac742f29d35b6a5f4841a83 192.168.188.130:8005@18005 myself,slave 8d96f3345386804e1f8bd66dbde956339a317ad0 0 1520618409000 5 connected
91a9ac78668ec9fe74d6768bc4ca4b39b38cdf49 192.168.188.130:8001@18001 master - 0 1520618413000 1 connected 333-5460
cc1c56c413f5ac2c42a87fe37502a7933e735a6d 192.168.188.130:8008@18008 slave 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483 0 1520618412000 7 connected
d2dd4adcadf9a76bcee03dd2ed2881cc980b93c1 192.168.188.130:8003@18003 master - 0 1520618410000 3 connected 11256-16383
f59a7d6b13d0a5a2b9e05db7c52e64945a43b8e9 192.168.188.130:8004@18004 slave 91a9ac78668ec9fe74d6768bc4ca4b39b38cdf49 0 1520618411000 4 connected
8d96f3345386804e1f8bd66dbde956339a317ad0 192.168.188.130:8002@18002 master - 0 1520618411064 2 connected 5795-10922

这里可能还会报错：

[ERR] Node 192.168.188.130:8001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

跟上面一样的步骤，删掉redis8目录下的dump.rdb、nodes.conf文件重启redis尝试

20、删除节点

① 直接删除会报错，如下：（后面是要删除节点的id编号）

[root@localhost redis-cluster]# ./redis-trib.rb del-node 192.168.188.130:8007 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483
>>> Removing node 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483 from cluster 192.168.188.130:8007
[ERR] Node 192.168.188.130:8007 is not empty! Reshard data away and try again.

② 将哈希槽分配出去，才能删除

[root@localhost redis-cluster]# ./redis-trib.rb reshard 192.168.188.130:8002
>>> Performing Cluster Check (using node 192.168.188.130:8002)
M: 8d96f3345386804e1f8bd66dbde956339a317ad0 192.168.188.130:8002
   slots:5795-10922 (5128 slots) master
   1 additional replica(s)
S: f59a7d6b13d0a5a2b9e05db7c52e64945a43b8e9 192.168.188.130:8004
   slots: (0 slots) slave
   replicates 91a9ac78668ec9fe74d6768bc4ca4b39b38cdf49
M: d2dd4adcadf9a76bcee03dd2ed2881cc980b93c1 192.168.188.130:8003
   slots:11256-16383 (5128 slots) master
   1 additional replica(s)
S: 719191fa7797cde14ac742f29d35b6a5f4841a83 192.168.188.130:8005
   slots: (0 slots) slave
   replicates 8d96f3345386804e1f8bd66dbde956339a317ad0
M: 91a9ac78668ec9fe74d6768bc4ca4b39b38cdf49 192.168.188.130:8001
   slots:333-5460 (5128 slots) master
   1 additional replica(s)
S: eeda69f0090ce204714d0b96d1e5565349a97a8a 192.168.188.130:8006
   slots: (0 slots) slave
   replicates d2dd4adcadf9a76bcee03dd2ed2881cc980b93c1
M: 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483 192.168.188.130:8007
   slots:0-332,5461-5794,10923-11255 (1000 slots) master
   1 additional replica(s)
S: cc1c56c413f5ac2c42a87fe37502a7933e735a6d 192.168.188.130:8008
   slots: (0 slots) slave
   replicates 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 1000
What is the receiving node ID? 8d96f3345386804e1f8bd66dbde956339a317ad0
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:377cb5f5e72e12d6c7cc6d90f3c3768f66f50483

这时会提示Source node#2，直接输入done就好了，槽将从8001移动到8002，刚好1000个

③ 再次查看8007已经没有槽了

M: 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483 192.168.188.130:8007
   slots: (0 slots) master
   0 additional replica(s)

④ 删除8007这个节点

[root@localhost redis-cluster]# ./redis-trib.rb del-node 192.168.188.130:8007 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483
>>> Removing node 377cb5f5e72e12d6c7cc6d90f3c3768f66f50483 from cluster 192.168.188.130:8007
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

⑤ 集群中已经没有8007这个节点了

192.168.188.130:8005> cluster nodes
eeda69f0090ce204714d0b96d1e5565349a97a8a 192.168.188.130:8006@18006 slave d2dd4adcadf9a76bcee03dd2ed2881cc980b93c1 0 1520619554501 6 connected
719191fa7797cde14ac742f29d35b6a5f4841a83 192.168.188.130:8005@18005 myself,slave 8d96f3345386804e1f8bd66dbde956339a317ad0 0 1520619551000 5 connected
91a9ac78668ec9fe74d6768bc4ca4b39b38cdf49 192.168.188.130:8001@18001 master - 0 1520619553000 1 connected 333-5460
cc1c56c413f5ac2c42a87fe37502a7933e735a6d 192.168.188.130:8008@18008 slave 8d96f3345386804e1f8bd66dbde956339a317ad0 0 1520619552000 8 connected
d2dd4adcadf9a76bcee03dd2ed2881cc980b93c1 192.168.188.130:8003@18003 master - 0 1520619552000 3 connected 11256-16383
f59a7d6b13d0a5a2b9e05db7c52e64945a43b8e9 192.168.188.130:8004@18004 slave 91a9ac78668ec9fe74d6768bc4ca4b39b38cdf49 0 1520619553000 4 connected
8d96f3345386804e1f8bd66dbde956339a317ad0 192.168.188.130:8002@18002 master - 0 1520619553494 8 connected 0-332 5461-11255

到这里redis集群就创建完成了，如果要停止集群直接到每个redis实例下面使用./redis-cli shutdown停止即可，或者使用如下脚本：

for a in 1 2 3 4 5 6 7 8
do
redis$a/redis-cli -p 800$a shutdown
done