redis集群搭建

1环境说明：
当前开启了四个节点，每个节点开三个redis实例，一共十二个redis，做成集群后为6主6从。ip和端口号分别如下：
192.168.1.63 port：16379、36379、37379
192.168.1.64 port：16379、36379、37379
192.168.1.65 port：16379、36379、37379
192.168.1.66 port：16379、36379、37379

2.安装redis:(在四个节点上都做下面的操作)
yum -y install gcc gcc-c++ epel-release wget vim
wget http://download.redis.io/releases/redis-5.0.4.tar.gz
tar xf redis-5.0.4.tar.gz
mv redis-5.0.4 /usr/local/redis
cd /usr/local/redis
make PREFIX=/usr/local/redis/ install
ln -s /usr/local/redis/bin/redis-* /usr/bin/
mkdir 16379 36379 37379
cp redis.conf 16379/
cp redis.conf 36379/
cp redis.conf 37379/

3.编辑配置文件:(四个节点都是相同操作)
vim 16379/redis.conf
bind 0.0.0.0
port 16379
daemonize yes
pidfile /var/run/redis_16379.pid
logfile “/var/log/16379.log”
dir /var/lib/16379
requirepass 123456
cluster-enabled yes
cluster-config-file nodes-16379.conf
cluster-node-timeout 15000

vim 36379/redis.conf
bind 0.0.0.0
port 36379
daemonize yes
pidfile /var/run/redis_36379.pid
logfile “/var/log/36379.log”
dir /var/lib/36379
requirepass 123456
cluster-enabled yes
cluster-config-file nodes-36379.conf
cluster-node-timeout 15000

vim 37379/redis.conf
bind 0.0.0.0
port 37379
daemonize yes
pidfile /var/run/redis_37379.pid
logfile “/var/log/37379.log”
dir /var/lib/37379
requirepass 123456
cluster-enabled yes
cluster-config-file nodes-37379.conf
cluster-node-timeout 15000

mkdir /var/lib/16379 /var/lib/36379 /var/lib/37379

4.安装ruby:(单台安装即可，后续在这台上面使用就行，yum安装的ruby版本太低，无法使用)
wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz
tar xf ruby-2.5.5.tar.gz
cd ruby-2.5.5
./configure
make
make install
ln -s /usr/local/bin/ruby /usr/bin/ruby
ruby -v

5.启动redis实例:(每台三个实例，注意带配置文件启动)
redis-server /usr/local/redis/16379/redis.conf &
redis-server /usr/local/redis/36379/redis.conf &
redis-server /usr/local/redis/37379/redis.conf &
ps -ef|grep redis

6.创建集群:(在安装ruby的那台节点上执行)
这里使用的是5.0版本的redis，创建集群命令都从./redis-trib.rb 迁移到redis-cli，可以使用redis-cli --cluster help来查看命令帮助；
redis-cli -a 123456 --cluster create 192.168.1.63:16379 192.168.1.63:37379 192.168.1.63:36379 192.168.1.64:16379 192.168.1.64:37379 192.168.1.64:36379 192.168.1.65:16379 192.168.1.65:37379 192.168.1.65:36379 192.168.1.66:16379 192.168.1.66:37379 192.168.1.66:36379 --cluster-replicas 1

执行后会出现以下信息
Master[0] -> Slots 0 - 2730
Master[1] -> Slots 2731 - 5460
Master[2] -> Slots 5461 - 8191
Master[3] -> Slots 8192 - 10922
Master[4] -> Slots 10923 - 13652
Master[5] -> Slots 13653 - 16383
Adding replica 192.168.1.66:37379 to 192.168.1.63:16379
Adding replica 192.168.1.63:36379 to 192.168.1.64:16379
Adding replica 192.168.1.64:36379 to 192.168.1.65:16379
Adding replica 192.168.1.65:36379 to 192.168.1.66:16379
Adding replica 192.168.1.66:36379 to 192.168.1.63:37379
Adding replica 192.168.1.65:37379 to 192.168.1.64:37379
**M: edea8633d85ebca09eeb7f63c4d06836ba2597c4 192.168.1.63:16379
slots:[0-2730] (2731 slots) master
M: 2642f7f37d7ff225f6bb83f38a5ba4805200288e 192.168.1.63:37379
slots:[10923-13652] (2730 slots) master
S: 3d9db359833cd0a45f536862be7946cc14e20724 192.168.1.63:36379
replicates d4ff7393e9b08b12f8355876cfda8e0fffdc9c6b
M: d4ff7393e9b08b12f8355876cfda8e0fffdc9c6b 192.168.1.64:16379
slots:[2731-5460] (2730 slots) master
M: 43bba2a6e127c0ac73ee9422dc1a300390c36644 192.168.1.64:37379
slots:[13653-16383] (2731 slots) master
S: 919a3d38fcd1e3e6462b7ad1ab1f2d9a5c1d724a 192.168.1.64:36379
replicates 42fa3954e82b1f57ba3deed58ebc600a7181e5b9
M: 42fa3954e82b1f57ba3deed58ebc600a7181e5b9 192.168.1.65:16379
slots:[5461-8191] (2731 slots) master
S: 200a602ba2d4d83c154d90520cbdf593bb48212e 192.168.1.65:37379
replicates 43bba2a6e127c0ac73ee9422dc1a300390c36644
S: ab0530c6f7f474d86f1e1fd51f1bcad495dbd3cc 192.168.1.65:36379
replicates 1f27d26e326f57dc382f77f495c6daa21021bdd1
M: 1f27d26e326f57dc382f77f495c6daa21021bdd1 192.168.1.66:16379
slots:[8192-10922] (2731 slots) master
S: debe53abbc92c6451128c4d146f733d33d385f66 192.168.1.66:37379
replicates edea8633d85ebca09eeb7f63c4d06836ba2597c4
S: efe7fa5beb18c0fe9c89f82d8724ae2d935bd38f 192.168.1.66:36379
replicates 2642f7f37d7ff225f6bb83f38a5ba4805200288e
Can I set the above configuration? (type ‘yes’ to accept): yes

Nodes configuration updated
Assign a different config epoch to each node
Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
…

Performing Cluster Check (using node 192.168.1.63:16379)
M: edea8633d85ebca09eeb7f63c4d06836ba2597c4 192.168.1.63:16379
slots:[0-2730] (2731 slots) master
1 additional replica(s)
S: 3d9db359833cd0a45f536862be7946cc14e20724 192.168.1.63:36379
slots: (0 slots) slave
replicates d4ff7393e9b08b12f8355876cfda8e0fffdc9c6b
M: 43bba2a6e127c0ac73ee9422dc1a300390c36644 192.168.1.64:37379
slots:[13653-16383] (2731 slots) master
1 additional replica(s)
M: 1f27d26e326f57dc382f77f495c6daa21021bdd1 192.168.1.66:16379
slots:[8192-10922] (2731 slots) master
1 additional replica(s)
M: d4ff7393e9b08b12f8355876cfda8e0fffdc9c6b 192.168.1.64:16379
slots:[2731-5460] (2730 slots) master
1 additional replica(s)
M: 42fa3954e82b1f57ba3deed58ebc600a7181e5b9 192.168.1.65:16379
slots:[5461-8191] (2731 slots) master
1 additional replica(s)
S: ab0530c6f7f474d86f1e1fd51f1bcad495dbd3cc 192.168.1.65:36379
slots: (0 slots) slave
replicates 1f27d26e326f57dc382f77f495c6daa21021bdd1
S: 200a602ba2d4d83c154d90520cbdf593bb48212e 192.168.1.65:37379
slots: (0 slots) slave
replicates 43bba2a6e127c0ac73ee9422dc1a300390c36644
M: 2642f7f37d7ff225f6bb83f38a5ba4805200288e 192.168.1.63:37379
slots:[10923-13652] (2730 slots) master
1 additional replica(s)
S: debe53abbc92c6451128c4d146f733d33d385f66 192.168.1.66:37379
slots: (0 slots) slave
replicates edea8633d85ebca09eeb7f63c4d06836ba2597c4
S: 919a3d38fcd1e3e6462b7ad1ab1f2d9a5c1d724a 192.168.1.64:36379
slots: (0 slots) slave
replicates 42fa3954e82b1f57ba3deed58ebc600a7181e5b9
S: efe7fa5beb18c0fe9c89f82d8724ae2d935bd38f 192.168.1.66:36379
slots: (0 slots) slave
replicates 2642f7f37d7ff225f6bb83f38a5ba4805200288e
[OK] All nodes agree about slots configuration.

Check for open slots…
Check slots coverage…
[OK] All 16384 slots covered.**

可以看到创建集群成功，并且主从已分如下：
Adding replica 192.168.1.66:37379 to 192.168.1.63:16379
Adding replica 192.168.1.63:36379 to 192.168.1.64:16379
Adding replica 192.168.1.64:36379 to 192.168.1.65:16379
Adding replica 192.168.1.65:36379 to 192.168.1.66:16379
Adding replica 192.168.1.66:36379 to 192.168.1.63:37379
Adding replica 192.168.1.65:37379 to 192.168.1.64:37379

192.168.1.66:37379是从，192.168.1.63:16379是主。
192.168.1.63:36379 是从，192.168.1.64:16379是主。
…如果没注意看到这个信息，也可以通过/var/log/16379.log等日志来找主从关系。
也就是说如果192.168.1.63:16379挂掉了，那么192.168.1.66:37379就会升级为主，代替192.168.1.63:16379来工作。

7.测试是否主挂掉了从实例自动转为主：
redis-cli --cluster info 192.168.1.63:16379 -a 123456
Warning: Using a password with ‘-a’ or ‘-u’ option on the command line interface may not be safe.
192.168.1.63:16379 (edea8633…) -> 0 keys | 2731 slots | 1 slaves.
192.168.1.63:36379 (3d9db359…) -> 1 keys | 2730 slots | 1 slaves.
192.168.1.64:37379 (43bba2a6…) -> 1 keys | 2731 slots | 1 slaves.
192.168.1.66:16379 (1f27d26e…) -> 0 keys | 2731 slots | 1 slaves.
192.168.1.65:16379 (42fa3954…) -> 1 keys | 2731 slots | 1 slaves.
192.168.1.63:37379 (2642f7f3…) -> 1 keys | 2730 slots | 1 slaves.
[OK] 4 keys in 6 masters.
0.00 keys per slot on average.

可以看到，每个主都有一个从，现在我们去把192.168.1.63:16379这个实例kill掉
redis-cli --cluster info 192.168.1.63:36379 -a 123456
Warning: Using a password with ‘-a’ or ‘-u’ option on the command line interface may not be safe.
Could not connect to Redis at 192.168.1.63:16379: Connection refused
192.168.1.63:36379 (3d9db359…) -> 1 keys | 2730 slots | 1 slaves.
192.168.1.64:37379 (43bba2a6…) -> 1 keys | 2731 slots | 1 slaves.
192.168.1.65:16379 (42fa3954…) -> 1 keys | 2731 slots | 1 slaves.
192.168.1.66:16379 (1f27d26e…) -> 0 keys | 2731 slots | 1 slaves.
192.168.1.66:37379 (debe53ab…) -> 0 keys | 2731 slots | 0 slaves.
192.168.1.63:37379 (2642f7f3…) -> 1 keys | 2730 slots | 1 slaves.
[OK] 4 keys in 6 masters.
0.00 keys per slot on average.

可以看到192.168.1.63:16379这个实例已经不在主的行列了，192.168.1.66:37379自动升级为主来到了主的行列，证明集群自动切换掉坏的主成功。

8.操作：（192.168.1.63上面操作）
[root@localhost redis]# redis-cli -c -p 36379 -a 123456
Warning: Using a password with ‘-a’ or ‘-u’ option on the command line interface may not be safe.
127.0.0.1:36379> KEYS *

1. “b”
   127.0.0.1:36379> set a 1
   -> Redirected to slot [15495] located at 192.168.1.64:37379
   OK
   192.168.1.64:37379> get a
   “1”
   192.168.1.64:37379>

可以看到自动切换到192.168.1.64:37379实例了，因为192.168.1.63：36379是从实例，无法写入。

9.集群添加新的节点
首选在192.168.1.63和192.168.1.64节点执行如下操作，新建一个配置文件
cd /usr/local/redis
cp -r 16379 27369
mkdir /var/lib/27369
vim 27369/redis.conf
把所有的16379替换成27369，然后wq保存退出

redis-cli --cluster add-node 192.168.1.63:27369 192.168.1.63:16379 -a 123456
192.168.1.63:27369为我们新的药添加的主节点，192.168.1.63:16379为集群中的任意节点

redis-cli --cluster info 192.168.1.63:16379 -a 123456
Warning: Using a password with ‘-a’ or ‘-u’ option on the command line interface may not be safe.
192.168.1.64:37379 (43bba2a6…) -> 2 keys | 2731 slots | 1 slaves.
192.168.1.66:37379 (debe53ab…) -> 0 keys | 2731 slots | 1 slaves.
192.168.1.63:37379 (2642f7f3…) -> 1 keys | 2730 slots | 1 slaves.
192.168.1.63:27369 (17a8f145…) -> 0 keys | 0 slots | 0 slaves.
192.168.1.66:16379 (1f27d26e…) -> 0 keys | 2731 slots | 1 slaves.
192.168.1.63:36379 (3d9db359…) -> 2 keys | 2730 slots | 1 slaves.
192.168.1.65:16379 (42fa3954…) -> 1 keys | 2731 slots | 1 slaves.
[OK] 6 keys in 7 masters.
0.00 keys per slot on average.

再次查看集群信息，可以看到192.168.1.63:27369已经在主节点行列了，还没有分配hash槽，我们还要为他分配hash槽。
redis-cli --cluster reshard 192.168.1.63:16379

192.168.1.63:16379为集群任意节点都行
然后出现一堆提示之后会让你选择添加的槽数，
填写你想添加的槽数然后回车。
然后会让你输入想要被分配槽的节点的id，id在上面信息中有
选择分配槽的方式是all还是done，我们选择all
分配成功
查看节点id：
redis-cli --cluster check 192.168.1.64:16379 -a 123456

再把192.168.1.64:27369加入到集群
redis-cli --cluster add-node 192.168.1.64:27369 192.168.1.63:16379 -a 123456

10.问题与注意事项：
问题1：…is not configured as a cluster node.

解决：没有开启配置： 修改 Clusster-enable yes

问题2：
*** FATAL CONFIG FILE ERROR *** Reading the configuration file, at line 23 >>> ‘slaveof 127.0.0.1 16370’ slaveof directive not allowed in cluster mode

解决：通过脚本启动集群配置，不允许一开始就配置脚本，会自己通过脚本来配置主从，注意脚本执行的参数

问题3： 配置文件中指定 cluster-config-file 时指定相关目录，启动失败，但没有提示，
redis-cluster-config目录异常导致启动失败，且没有相关的提示

解决：此处指定的文件都是基于 dir 的配置，直接配置文件名就好
Tips: pid 文件、log文件都不是基于dir的，是基于执行的根目录，而aof是基于的

问题4：集群端口除了基本的外部端口外，还有一个内部各实例的通信端口，基于外部端口+10000，即：原为：16379，则内部实例通信端口为：26379，注意不在被占用了
Redis cluster 启动会占用两个端口，容易导致链接失败

解决：修改端口；Redis集群中每个redis实例（可能一台机部署多个实例）会使用两个Tcp端口，一个用于给客户端（redis-cli或应用程序等）使用的端口，另一个是用于集群中实例相互通信的内部总线端口，且第二个端口比第一个端口一定大10000.内部总线端口通信使用特殊协议，以便实现集群内部高带宽低时延的数据交换。

问题5：redis-cli -p 6370
redis-cli 默认是不支持集群服务器的，报错：(error) MOVED 5798 127.0.0.1:6371

解决：
redis-cli -p 6370 -c # 添加参数，使用Redis默认的集群客户端

问题6：Sorry, the cluster configuration file nodes.conf is already used bySorry, the cluster configuration file nodes.conf is already used by

解决：cluster-config-file没有修改，导致重复，修改配置文件的该项，保证每个实例该配置不重复。

问题7：[ERR] Node 127.0.0.1:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

解决：提示我们redis服务实例上的数据不为空，因此逐个实例连接，执行flushdb，清空数据。

问题8：[ERR] Not all 16384 slots are covered by nodes.

解决：提示我们不是所有的slot都被节点覆盖到，官方的建议是使用fix修复。执行redis-cli --cluster fix,显示了修正结果，槽位重新分配。

问题9：如果创建集群失败，需要关闭所有节点，找到nodes-port.conf文件并删除，然后重新执行创建集群命令。