Redis集群模式搭建

简介：Redis Cluster是Redis的分布式解决方案，有效解决了Redis分布式方面的需求。当遇到单机内存、并发、流量等瓶颈时，可以采用Cluster架构达到负载均衡的目的。

集群搭建需要的环境

- Redis集群至少需要3个节点。因为投票容错机制要求超过半数节点认为某个节点挂了该节点才是挂了，所以2个节点无法构成集群。
- 要保证集群的高可用，需要每个节点都有从节点，也就是备份节点，所以Redis集群至少需要6台服务器。

因此用两台虚拟机模拟6个节点，一台机器3个节点，创建出3 master、3 salve环境。
两台虚拟机都是 CentOS 7，ip分别为192.168.186.131 和 192.168.186.132。

一、安装过程

1.下载并解压

cd /usr/local/src/
wget http://download.redis.io/releases/redis-4.0.11.tar.gz
tar -zxvf redis-4.0.11.tar.gz

2.编译安装

cd redis-4.0.11
make
make install PREFIX=/usr/local/redis  (PREFIX指定安装目录)

二、构建ruby环境

搭建redis集群需要使用一个ruby工具（redis-trib.rb），因为这个工具是一个ruby脚本文件，所以这个工具的运行需要ruby的运行环境。

• 下载安装ruby

yum -y install ruby ruby-devel rubygems rpm-build

• 把ruby相关的包安装到服务器，我这里用的是redis-3.0.0.gem

gem install redis-3.0.0.gem

三、集群搭建步骤（关闭防火墙）

1. 首先在 192.168.186.131 机器的usr/local/redis目录下新建cluster目录，用于存放集群节点。
2. 在cluster目录下，创建名为7001、7002、7003的目录，并将 redis.conf 拷贝到这三个目录中。
3. 分别修改这三个配置文件，修改如下内容

port 7001                             //端口7001,7002,7003        
bind 192.168.186.131                  //默认ip为127.0.0.1 需要改为其他节点机器可访问的ip 否则创建集群时无法访问对应的端口，无法创建集群
daemonize yes                         //后台运行
pidfile /var/run/redis_7001.pid       //pidfile文件对应7001,7002,7003
logfile "7001.log"					  //生成节点服务器日志
cluster-enabled yes                   //开启集群  把注释#去掉
cluster-config-file nodes_7001.conf   //集群的配置  配置文件首次启动自动生成 7001,7002,7003
cluster-node-timeout 15000            //请求超时  默认15秒，可自行设置

4. 接着在另外一台机器上 192.168.186.132，操作重复以上三步，把目录改为7004、7005、7006，对应的配置文件也按照这个规则修改即可。
5. 启动各个节点（可编写脚本启动）

第一台机器192.168.186.131上执行
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7001/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7002/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7003/redis.conf

第二台机器192.168.186.132上执行
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7004/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7005/redis.conf
/usr/local/redis/bin/redis-server /usr/local/redis/cluster/7006/redis.conf

6. 检查redis启动情况

第一台机器</br>
# ps -ef | grep redis
root      63768      1  0 14:49 ?        00:00:05 /usr/local/redis/bin/redis-server 192.168.186.131:7001 [cluster]
root      63770      1  0 14:49 ?        00:00:04 /usr/local/redis/bin/redis-server 192.168.186.131:7002 [cluster]
root      63772      1  0 14:49 ?        00:00:05 /usr/local/redis/bin/redis-server 192.168.186.131:7003 [cluster]

# netstat -tnlp | grep redis
tcp        0      0 192.168.186.131:17001   0.0.0.0:*               LISTEN      63768/redis-server  
tcp        0      0 192.168.186.131:17002   0.0.0.0:*               LISTEN      63770/redis-server  
tcp        0      0 192.168.186.131:17003   0.0.0.0:*               LISTEN      63772/redis-server  
tcp        0      0 192.168.186.131:7001    0.0.0.0:*               LISTEN      63768/redis-server  
tcp        0      0 192.168.186.131:7002    0.0.0.0:*               LISTEN      63770/redis-server  
tcp        0      0 192.168.186.131:7003    0.0.0.0:*               LISTEN      63772/redis-server 

--------------------------------------------------------------------------------------------------------------------

第二台机器</br>
# ps -ef | grep redis
root      67525      1  0 14:49 ?        00:00:05 /usr/local/redis/bin/redis-server 192.168.186.132:7004 [cluster]
root      67527      1  0 14:49 ?        00:00:05 /usr/local/redis/bin/redis-server 192.168.186.132:7005 [cluster]
root      67532      1  0 14:49 ?        00:00:05 /usr/local/redis/bin/redis-server 192.168.186.132:7006 [cluster]

# netstat -tnlp | grep redis
tcp        0      0 192.168.186.132:17004   0.0.0.0:*               LISTEN      67525/redis-server  
tcp        0      0 192.168.186.132:17005   0.0.0.0:*               LISTEN      67527/redis-server  
tcp        0      0 192.168.186.132:17006   0.0.0.0:*               LISTEN      67532/redis-server  
tcp        0      0 192.168.186.132:7004    0.0.0.0:*               LISTEN      67525/redis-server  
tcp        0      0 192.168.186.132:7005    0.0.0.0:*               LISTEN      67527/redis-server  
tcp        0      0 192.168.186.132:7006    0.0.0.0:*               LISTEN      67532/redis-server

7. 创建集群

Redis 官方提供了 redis-trib.rb 这个工具，就在解压目录的 src 目录中。可将它复制到 /usr/local/redis/bin 目录中，用于创建集群。

redis-trib.rb  create  --replicas  1 192.168.186.131:7001 192.168.186.131:7002 192.168.186.131:7003 192.168.186.132:7004 192.168.186.132:7005  192.168.186.132:7006 

其中，前三个为第一台机器的节点，其余三个为第二台机器的节点。
执行 redis-trib.rb 命令，会出现如下提示：

>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.186.131:7001
192.168.186.132:7004
192.168.186.131:7002
Adding replica 192.168.186.132:7006 to 192.168.186.131:7001
Adding replica 192.168.186.131:7003 to 192.168.186.132:7004
Adding replica 192.168.186.132:7005 to 192.168.186.131:7002
M: 8635b4b0a3e5bbeb628fc570156946936ba7da80 192.168.186.131:7001
   slots:0-5460 (5461 slots) master
M: 72c23b778c9f14869883336d0b35740fedf2e00a 192.168.186.131:7002
   slots:10923-16383 (5461 slots) master
S: 6ac87210235f4a0ed4cbaafd6a22f4558d60df3a 192.168.186.131:7003
   replicates 93d81024b10ea6299e560d7dd4e9035fb0fbfa65
M: 93d81024b10ea6299e560d7dd4e9035fb0fbfa65 192.168.186.132:7004
   slots:5461-10922 (5462 slots) master
S: 3dcf2f0a2f45b1780ff2573ac17f3af5191eee95 192.168.186.132:7005
   replicates 72c23b778c9f14869883336d0b35740fedf2e00a
S: d84f9c05667e3c882a4f09a6f53670464f51e896 192.168.186.132:7006
   replicates 8635b4b0a3e5bbeb628fc570156946936ba7da80
Can I set the above configuration? (type 'yes' to accept): yes

输入 yes 即可，然后出现如下内容，说明安装成功。

>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.186.131:7001
192.168.186.132:7004
192.168.186.131:7002
Adding replica 192.168.186.132:7006 to 192.168.186.131:7001
Adding replica 192.168.186.131:7003 to 192.168.186.132:7004
Adding replica 192.168.186.132:7005 to 192.168.186.131:7002
M: 8635b4b0a3e5bbeb628fc570156946936ba7da80 192.168.186.131:7001
   slots:0-5460 (5461 slots) master
M: 72c23b778c9f14869883336d0b35740fedf2e00a 192.168.186.131:7002
   slots:10923-16383 (5461 slots) master
S: 6ac87210235f4a0ed4cbaafd6a22f4558d60df3a 192.168.186.131:7003
   replicates 93d81024b10ea6299e560d7dd4e9035fb0fbfa65
M: 93d81024b10ea6299e560d7dd4e9035fb0fbfa65 192.168.186.132:7004
   slots:5461-10922 (5462 slots) master
S: 3dcf2f0a2f45b1780ff2573ac17f3af5191eee95 192.168.186.132:7005
   replicates 72c23b778c9f14869883336d0b35740fedf2e00a
S: d84f9c05667e3c882a4f09a6f53670464f51e896 192.168.186.132:7006
   replicates 8635b4b0a3e5bbeb628fc570156946936ba7da80
Can I set the above configuration? (type 'yes' to accept): yes

8. 集群验证

• 在第一台机器上连接集群的7002端口的节点，在另一台机器上连接7005节点，连接方式为：

/usr/local/redis/bin/redis-cli -h 192.168.186.131 -p 7002 -c  (-c 参数可连接到集群，不然节点之间是无法自动跳转的)

• 在7002节点执行命令，执行结果如下：

192.168.186.131:7002> set hello world
-> Redirected to slot [866] located at 192.168.186.131:7001
OK
192.168.186.131:7001>

• 连接7005端口，查看 key 为 hello 的内容，执行结果如下：

192.168.186.132:7005> get hello
-> Redirected to slot [866] located at 192.168.186.131:7001
"world"
192.168.186.131:7001> 

到此，说明redis集群运行正常。

四、填坑

在搭建redis集群过程中踩过不少坑，对一些常见错误进行总结归纳。

1.[ERR] Sorry, can’t connect to node

[root@pengxianghui bin]# ./redis-trib.rb  create  --replicas  1 192.168.186.131:7001 192.168.186.131:7002 192.168.186.131:7003 192.168.186.132:7004 192.168.186.132:7005  192.168.186.132:7006 
>>> Creating cluster
[ERR] Sorry, can't connect to node 192.168.186.132:7004

解决办法：关闭防火墙

• 查看防火墙状态 ：systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since 六 2019-06-29 13:37:05 CST; 1h 16min ago
     Docs: man:firewalld(1)
 Main PID: 6358 (firewalld)
    Tasks: 2
   CGroup: /system.slice/firewalld.service
           └─6358 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

• 关闭防火墙：

方法1：systemctl stop firewalld（事实证明，只执行这个，重启之后是不行的，还必须执行systemclt disable firewalld）

[root@pengxianghui bin]# systemctl stop firewalld
[root@pengxianghui bin]# systemctl disable firewalld
[root@pengxianghui bin]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

方法2：编辑文件 /etc/selinux/config，将selinux的值改成disable，然后重启生效（reboot）

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted