redis集群（2） 添加删除节点

前言：

生产环境中单节点redis服务器带来的问题

• 单点故障，导致服务不可以正常使用

• 无法处理大量的并发数据请求

• 容易造成数据丢失

解决办法：

• 搭建redis集群

一 ：redis集群介绍

redis集群是一个提供在多个redis节点间共享数据的程序集

redis集群并不支持同时处理多个keys的命令，因为keys存放在不同节点，待处理的keys不一定同时在一台redis节点上；这需要在不同节点间移动数据，从而达不到向redis那样的性能，在高负载的情况下可能会导致不可预料的错误

redis集群通过分区来提供一定程度的可用性，在实际环境中，当某个节点宕机或不可达的情况下继续处理命令

二：redis集群的优势

自动分割数据到不同的节点上

整个集群的部分节点失败或者不可达的情况下能够继续处理命令

三：redis集群的数据分片

redis集群没有使用一致性hash，而是引入了哈希槽的概念

redis集群有16384个哈希槽

每个key通过crc16校验后对16384取模来决定放置哪个槽

集群的每个节点负责一部分hash槽

3.1 以3个节点组成的集群为例

节点A包含0-5500号哈希槽

节点B包含5501-11000号哈希槽

节点C包含11001-16383号哈希槽

3.2 支持添加或者删除节点

添加删除节点无需停止服务

比如：

我想添加节点D，我需要移动节点A、B、C中的部分槽到D上；

我想一处节点A，我需要将A中的槽移动到B和C节点上，然后将没有任何槽的节点A从集群中一处

四：redis集群中的主从复制模型

集群中具有A、B、C三个基点，如果B故障，整个集群就会因缺少5501-11000这个范围的哈希槽而导致不可用

• 为每个节点添加一个从节点，A1、B1、C1，整个集群便有三个master节点和三个slave节点组成，在节点B发生故障后，集群选择B1为新的主节点继续服务
• 当B和B1都失效后，集群将不可用

五：实操

操作思路

1、安装Redis
2、修改配置文件
3、使用脚本创建集群
4、测试集群

5.1 修改主机名，设置网卡

节点A，IP192.168.247.155

[root@lamp ~]# hostnamectl set-hostname A
[root@lamp ~]# su
[root@a ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.247.155  netmask 255.255.255.0  broadcast 192.168.247.255

从节点A1，IP192.168.247.160

[root@k8s-node01 ~]# hostnamectl set-hostname a1
[root@k8s-node01 ~]# su
[root@a1 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.247.160  netmask 255.255.255.0  broadcast 192.168.247.255

节点B，IP192.168.247.134

[root@b ~]# ip addr
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:03:f6:78 brd ff:ff:ff:ff:ff:ff
    inet 192.168.247.134/24 brd 192.168.247.255 scope global dynamic ens32

从节点B1，IP192.168.247.20

[root@ct ~]# hostnamectl set-hostname b1
[root@ct ~]# su
[root@b1 ~]# ip addr
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:97:36:a9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.247.20/24 brd 192.168.247.255 scope global eth1

5.2 安装装机必备组件

备注：搭建本地仓库，或者使用现网yum源，记得配置dns
本次使用下载现网源的方法

[root@a yum.repos.d]# curl -o /etc/yum.repos.d/CentOS.repo http://mirrors.aliyun.com/repo/Centos-7.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2523  100  2523    0     0   1634      0  0:00:01  0:00:01 --:--:--  1635
[root@a yum.repos.d]# yum makecache

[root@a ~]# yum install -y gcc gcc-c++ make vim bash-completion net-tools samba cifs-utils samba-client

5.3 将redis-5.0.7.tar.gz 上传到各个节点

使用mout.cifs挂在宿主机的共享文件夹，如何将宿主机的文件夹共享出去看我的博客
https://blog.csdn.net/Lfwthotpt/article/details/102834201

[root@b ~]# mkdir /abc
[root@b ~]# mount.cifs //192.168.0.88/linuxs /abc -o vers=2.0
-bash: mount.cifs: command not found
[root@b ~]# yum install -y samba cifs-utils samba-client
[root@a ~]# mount.cifs //192.168.0.88/linuxs /abc -o vers=2.0
Password for root@//192.168.0.88/linuxs:  
[root@b ~]# cd /abc
[root@b abc]# tar zxvf redis-5.0.7.tar.gz -C /opt

各个节点关闭防火墙和核心防护

[root@a ~]# systemctl stop firewalld
[root@a ~]# systemctl disable firewalld
[root@a ~]# setenforce 0
[root@a ~]# sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

5.4 各个节点都编译安装

[root@b abc]# cd /opt/redis-5.0.7/
[root@b redis-5.0.7]# ls
00-RELEASENOTES  CONTRIBUTING  deps     Makefile   README.md   runtest          runtest-moduleapi  sentinel.conf  tests
BUGS             COPYING       INSTALL  MANIFESTO  redis.conf  runtest-cluster  runtest-sentinel   src 
[root@b redis-5.0.7]# make
[root@b redis-5.0.7]# make PREFIX=/usr/local/redis install		//指定安装路径

5.5 设置Redis相关配置文件

[root@b redis-5.0.7]# ln -s /usr/local/redis/bin/* /usr/local/bin/			//优化命令路径
[root@b redis-5.0.7]# cd /opt/redis-5.0.7/utils/
[root@b utils]# ./install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server
Please select the redis port for this instance: [6379] 		//默认即可，回车
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf] 			//默认即可，回车
Selected default - /etc/redis/6379.conf
Please select the redis log file name [/var/log/redis_6379.log] 			//默认即可，回车
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379] 			//默认即可，回车
Selected default - /var/lib/redis/6379
Please select the redis executable path [/usr/local/bin/redis-server] /usr/local/bin/redis-server		
//注意，此处直接回车不会生成该客户端命令，需要手动输入
Selected config:
Port           : 6379					//端口号
Config file    : /etc/redis/6379.conf				//配置文件
Log file       : /var/log/redis_6379.log			//日志文件
Data dir       : /var/lib/redis/6379					//数据目录
Executable     : /usr/local/bin/redis-server		//执行命令
Cli Executable : /usr/local/bin/redis-cli			//客户端命令
Is this ok? Then press ENTER to go on or Ctrl-C to abort.		//回车
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
/var/run/redis_6379.pid exists, process is already running or crashed
Installation successful!
[root@b utils]# netstat -natp | grep 6379			//现在已经开启
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      84062/redis-server  

5.6 修改各个redis主机的6379.conf配置文件

[root@b utils]# vim /etc/redis/6379.conf 
  70 #bind 127.0.0.1			//注释掉代表监听所有
  89 protected-mode no		//不开启保护功能
  93 port 6379						//默认为6379
 137 daemonize yes			//以独立进程启动
 172 logfile /var/log/redis_6379.log		//默认日志文件路径
 700 appendonly yes			//开启aof同步
 833 cluster-enabled yes		//开启群集
 841 cluster-config-file nodes-6379.conf		//指定群集配置文件
 847 cluster-node-timeout 15000		//群集超时时间设置，单位毫秒
 924 cluster-require-full-coverage no
 //表示当负责一个插槽的主库下线且没有相应的从库进行故障恢复时，集群仍然可用

5.7 各个节点重启redis

[root@a utils]# /etc/init.d/redis_6379 restart
Stopping ...
Waiting for Redis to shutdown ...
Redis stopped
Starting Redis server...

查看生成的redis集群配置文件

[root@a utils]# cd /var/lib/redis/6379/
[root@a 6379]# ls
appendonly.aof  dump.rdb  nodes-6379.conf
[root@a 6379]# netstat -antp |grep  6379
tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      92266/redis-server  
tcp        0      0 0.0.0.0:16379           0.0.0.0:*               LISTEN      92266/redis-server  
tcp6       0      0 :::6379                 :::*                    LISTEN      92266/redis-server  
tcp6       0      0 :::16379                :::*                    LISTEN      92266/redis-server 

5.8 创建集群，在a上面做就可以

[root@a ~]#  yum -y install ruby rubygems 
[root@a ~]# cd /abc
[root@a abc]# cp redis-3.2.0.gem ~/
[root@a abc]# cd ~
[root@a ~]# gem install redis --version 3.2.0
Successfully installed redis-3.2.0
Parsing documentation for redis-3.2.0
Installing ri documentation for redis-3.2.0
1 gem installed

切换到安装包的src/下

[root@a ~]# cd /opt/redis-5.0.7/src/
[root@a src]# redis-cli --cluster create --cluster-replicas 1 \
> 192.168.247.155:6379 \
> 192.168.247.134:6379 \
> 192.168.247.160:6379 \
> 192.168.247.20:6379
*** ERROR: Invalid configuration for cluster creation.
*** Redis Cluster requires at least 3 master nodes.
*** This is not possible with 4 nodes and 1 replicas per node.
*** At least 6 nodes are required.

出现报错，反馈至少六个节点
再创建两台redis
c1 IP192.168.247.136

[root@comp2 ~]# hostnamectl set-hostname c1
[root@comp2 ~]# su
[root@c1 ~]# ip addr
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:3e:08:70 brd ff:ff:ff:ff:ff:ff
    inet 192.168.247.136/24 brd 192.168.247.255 scope global dynamic eth1

c IP 192.168.247.135

[root@comp1 ~]# hostnamectl set-hostname c
[root@comp1 ~]# su
[root@c ~]# ip addr
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:56:b1:70 brd ff:ff:ff:ff:ff:ff
    inet 192.168.247.135/24 brd 192.168.247.255 scope global dynamic eth1

redis配置完毕后，重新设置
中间记得输入yes

[root@a src]# redis-cli --cluster create --cluster-replicas 1 192.168.247.155:6379 192.168.247.134:6379 192.168.247.160:6379 192.168.247.20:6379 192.168.247.135:6379 192.168.247.136:6379
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460		//第一个mater的哈希槽区间
Master[1] -> Slots 5461 - 10922		//第二个mater的哈希槽区间
Master[2] -> Slots 10923 - 16383		第三个mater的哈希槽区间
Adding replica 192.168.247.135:6379 to 192.168.247.155:6379		//从下面的数据看，155是master，对应的slave是135
Adding replica 192.168.247.136:6379 to 192.168.247.134:6379		//134是master，对应的slave是136
Adding replica 192.168.247.20:6379 to 192.168.247.160:6379			//160是master，对应的slave是20
M: 28669db5a8328297b270018e4ef84618a2a0da9a 192.168.247.155:6379
   slots:[0-5460] (5461 slots) master
M: c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379
   slots:[5461-10922] (5462 slots) master
M: 0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379
   slots:[10923-16383] (5461 slots) master
S: af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379
   replicates 0d257e1d225c2e03bc54dbf128112ed8f144be82
S: 4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379
   replicates 28669db5a8328297b270018e4ef84618a2a0da9a
S: 187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379
   replicates c802859b1ebdfca96fcbe0ec48609cc8dd171dd2
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.....
>>> Performing Cluster Check (using node 192.168.247.155:6379)
M: 28669db5a8328297b270018e4ef84618a2a0da9a 192.168.247.155:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379
   slots: (0 slots) slave
   replicates c802859b1ebdfca96fcbe0ec48609cc8dd171dd2
S: af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379
   slots: (0 slots) slave
   replicates 0d257e1d225c2e03bc54dbf128112ed8f144be82
M: c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379
   slots: (0 slots) slave
   replicates 28669db5a8328297b270018e4ef84618a2a0da9a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

5.9 检查集群状态

主从节点关系是随机分配

[root@a src]# redis-cli --cluster check 192.168.247.155:6379
192.168.247.155:6379 (28669db5...) -> 0 keys | 5461 slots | 1 slaves.
192.168.247.134:6379 (c802859b...) -> 0 keys | 5462 slots | 1 slaves.
192.168.247.160:6379 (0d257e1d...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.247.155:6379)
M: 28669db5a8328297b270018e4ef84618a2a0da9a 192.168.247.155:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379
   slots: (0 slots) slave
   replicates c802859b1ebdfca96fcbe0ec48609cc8dd171dd2
S: af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379
   slots: (0 slots) slave
   replicates 0d257e1d225c2e03bc54dbf128112ed8f144be82
M: c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379
   slots: (0 slots) slave
   replicates 28669db5a8328297b270018e4ef84618a2a0da9a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

六：测试集群

6.1 查看集群信息

[root@a src]# redis-cli -h 192.168.247.155 -p 6379 -c
192.168.247.155:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:922
cluster_stats_messages_pong_sent:935
cluster_stats_messages_sent:1857
cluster_stats_messages_ping_received:930
cluster_stats_messages_pong_received:922
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1857

6.2 查看节点信息

192.168.247.155:6379> cluster nodes
187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379@16379 slave c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 0 1584861111000 6 connected
28669db5a8328297b270018e4ef84618a2a0da9a 192.168.247.155:6379@16379 myself,master - 0 1584861111000 1 connected 0-5460
af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379@16379 slave 0d257e1d225c2e03bc54dbf128112ed8f144be82 0 1584861113000 3 connected
c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379@16379 master - 0 1584861113131 2 connected 5461-10922
0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379@16379 master - 0 1584861114138 3 connected 10923-16383
4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379@16379 slave 28669db5a8328297b270018e4ef84618a2a0da9a 0 1584861113000 5 connected

master	slave
192.168.247.155 a	192.168.247.135 c
192.168.247.134 b	192.168.247.136 c1
192.168.247.160 a1	192.168.247.20 b1

6.3 验证数据

• 在155master上创建一个user-lisi，被保存到134master上，并且自动切换到134master上

[root@a src]# redis-cli -h 192.168.247.155 -p 6379 -c
192.168.247.155:6379> keys *
(empty list or set)
192.168.247.155:6379> set user lisi
-> Redirected to slot [5474] located at 192.168.247.134:6379		//反馈数据被保存到134节点
OK
192.168.247.134:6379> keys *
1) "user"
192.168.247.134:6379> get user
"lisi"
192.168.247.134:6379> exit

• 在155master上再次创建qq-gsy，依旧被保存到134master上
• 切换到155master，keys * 查看，发现155redis数据库内并没有数据

[root@a src]# redis-cli -h 192.168.247.155 -p 6379 -c
192.168.247.155:6379> set qq gsy
-> Redirected to slot [5598] located at 192.168.247.134:6379
OK
192.168.247.134:6379> exit
[root@a src]# redis-cli -h 192.168.247.134 -p 6379 -c
192.168.247.134:6379> keys *
1) "user"
2) "qq"
192.168.247.134:6379> exit
[root@a src]# redis-cli -h 192.168.247.155 -p 6379 -c
192.168.247.155:6379> keys *
(empty list or set)

• 再次在155master节点创建155-1，此时数据被保存到160master上，并且数据库自动切换到160master上
• 切换到135slave（155的从节点），没有数据，get 155，获取值并且自动切换到160master上

192.168.247.155:6379> set 155 1
-> Redirected to slot [11779] located at 192.168.247.160:6379		//数据被保存到160节点
OK
192.168.247.160:6379> keys *
1) "155"
192.168.247.160:6379> quit
[root@a src]# redis-cli -h 192.168.247.135 -p 6379 -c
192.168.247.135:6379> keys *
(empty list or set)
192.168.247.135:6379> get 155
-> Redirected to slot [11779] located at 192.168.247.160:6379
"1"
192.168.247.160:6379> keys *
1) "155"
192.168.247.160:6379> quit

• 进入134 master上，查看到之前在155上时创建的键值
• 切换到134的从节点136，键值备份
• 20slave是160master的从节点，里面有在155master时创建的155-1的备份

[root@a src]# redis-cli -h 192.168.247.134 -p 6379 -c
192.168.247.134:6379> keys *
1) "user"
2) "qq"
[root@a src]# redis-cli -h 192.168.247.136 -p 6379 -c
192.168.247.136:6379> keys *
1) "qq"
2) "user"
192.168.247.136:6379> quit
[root@a src]# redis-cli -h 192.168.247.20 -p 6379 -c
192.168.247.20:6379> keys *
1) "155"

七：增减节点测试

在实际的生产环境中，由于业务大规模升级，项目中的前置缓存节点，已经不能满足当前的业务需求，
现在对redis集群扩容一套redis主备节点，对于实际的生产环境，一般规划是在晚上2凌晨12点做操作
操作之前，提前做好redis安装部署，晚上凌晨12点左右，业务量比较少，影响范围比较少
redis对接的底层数据库，需要先要锁库、锁表，不产生业务流数据，这样redis集群可产生的影响因素较小。

7.1 删掉155节点

先创建一个在155上面保存的数据

[root@a ~]# redis-cli -h 192.168.247.134 -p 6379 -c
192.168.247.134:6379> set 134 2
-> Redirected to slot [5252] located at 192.168.247.155:6379
OK
192.168.247.155:6379> keys *
1) "134"
192.168.247.155:6379> quit
[root@a ~]# cd /var/lib/redis/6379/
[root@a 6379]# ls
appendonly.aof  dump.rdb  nodes-6379.conf
[root@a 6379]# ll
total 12
-rw-r--r--. 1 root root  52 Mar 22 15:46 appendonly.aof
-rw-r--r--. 1 root root 187 Mar 22 15:46 dump.rdb
-rw-r--r--. 1 root root 816 Mar 22 14:55 nodes-6379.conf
[root@a 6379]# init 0

155有一个数据，134-2

7.2 查看此时的redis集群节点信息

之前在155的数据没有丢失

[root@c ~]# redis-cli -h 192.168.247.160 -p 6379 -c
192.168.247.160:6379> get 134
-> Redirected to slot [5252] located at 192.168.247.135:6379
"2"
192.168.247.135:6379> cluster nodes
28669db5a8328297b270018e4ef84618a2a0da9a 192.168.247.155:6379@16379 master,fail - 1584863259988 1584863258581 1 connected
0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379@16379 master - 0 1584863564000 3 connected 10923-16383
187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379@16379 slave c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 0 1584863564916 6 connected
af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379@16379 slave 0d257e1d225c2e03bc54dbf128112ed8f144be82 0 1584863562000 4 connected
c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379@16379 master - 0 1584863563000 2 connected 5461-10922
4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379@16379 myself,master - 0 1584863561000 7 connected 0-5460

删除一个master没问题，135就是155的slave
之前155master显示失败，可以把它从集群中剔除

7.3 在剔除失效节点之前，重新创建一个redis，先加入其中，代替155

abei IP 192.168.247.133

[root@localhost ~]# hostnamectl set-hostname abei
[root@localhost ~]# su
[root@abei ~]# ip addr
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    inet 192.168.247.133/24 brd 192.168.247.255 scope global dynamic ens32

部署redis见上文

部署完毕，加入到集群中

[root@c ~]# redis-cli --cluster add-node 192.168.247.133:6379 192.168.247.135:6379 		
//redis-cli --cluster add-node 新节点IP:端口号 当前集群中已存在的任意节点IP：端口号		
>>> Adding node 192.168.247.133:6379 to cluster 192.168.247.135:6379
Could not connect to Redis at 192.168.247.155:6379: No route to host
>>> Performing Cluster Check (using node 192.168.247.135:6379)
M: 4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379
   slots:[0-5460] (5461 slots) master
M: 0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379
   slots: (0 slots) slave
   replicates c802859b1ebdfca96fcbe0ec48609cc8dd171dd2
S: af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379
   slots: (0 slots) slave
   replicates 0d257e1d225c2e03bc54dbf128112ed8f144be82
M: c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.247.133:6379 to make it join the cluster.
[OK] New node added correctly.		//反馈新加入一个节点

7.1 新加入的节点都是主节点的身份，但是没有哈希槽，不可用

[root@c ~]# redis-cli --cluster check 192.168.247.135:6379
Could not connect to Redis at 192.168.247.155:6379: No route to host
192.168.247.135:6379 (4f4175a5...) -> 1 keys | 5461 slots | 0 slaves.
192.168.247.133:6379 (0e3f5fed...) -> 0 keys | 0 slots | 0 slaves.		//新加入的节点
192.168.247.160:6379 (0d257e1d...) -> 1 keys | 5461 slots | 1 slaves.
192.168.247.134:6379 (c802859b...) -> 2 keys | 5462 slots | 1 slaves.
[OK] 4 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.247.135:6379)
M: 4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379
   slots:[0-5460] (5461 slots) master
M: 0e3f5fed661b7c92ccfac14cb5c1a24959aa61bd 192.168.247.133:6379
   slots: (0 slots) master
M: 0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379
   slots: (0 slots) slave
   replicates c802859b1ebdfca96fcbe0ec48609cc8dd171dd2
S: af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379
   slots: (0 slots) slave
   replicates 0d257e1d225c2e03bc54dbf128112ed8f144be82
M: c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

7.2 数据库内查看集群节点信息

155fail
133没啥用

192.168.247.135:6379> cluster nodes
0e3f5fed661b7c92ccfac14cb5c1a24959aa61bd 192.168.247.133:6379@16379 master - 0 1584865919035 0 connected
28669db5a8328297b270018e4ef84618a2a0da9a 192.168.247.155:6379@16379 master,fail - 1584863259988 1584863258581 1 connected
0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379@16379 master - 0 1584865916014 3 connected 10923-16383
187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379@16379 slave c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 0 1584865918029 6 connected
af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379@16379 slave 0d257e1d225c2e03bc54dbf128112ed8f144be82 0 1584865917021 4 connected
c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379@16379 master - 0 1584865916000 2 connected 5461-10922
4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379@16379 myself,master - 0 1584865916000 7 connected 0-5460

7.3 删除节点133，再次添加

redis-cli --cluster del-node 集群任意目标节点IP:端口号 要删除的节点ID

[root@c ~]# redis-cli --cluster del-node 192.168.247.133:6379 0e3f5fed661b7c92ccfac14cb5c1a24959aa61bd 
//redis-cli --cluster del-node 集群任意目标节点IP:端口号 要删除的节点ID
>>> Removing node 0e3f5fed661b7c92ccfac14cb5c1a24959aa61bd from cluster 192.168.247.133:6379
Could not connect to Redis at 192.168.247.155:6379: No route to host
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

[root@c src]# redis-cli --cluster add-node 192.168.247.133:6379 192.168.247.135:6379
[ERR] Node 192.168.247.133:6379 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

发现报错了
找到一个解决办法
到出错节点内，删掉关于redis集群的记录，然后重启
查看当前数据库内有无数据，有的话就全部 flushdb删除掉

[root@abei ~]# cd /var/lib/redis/6379/
[root@abei 6379]# ls
appendonly.aof  dump.rdb  nodes-6379.conf
[root@abei 6379]# rm -rf *
[root@abei 6379]# ls
[root@abei 6379]# /etc/init.d/redis_6379 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@abei 6379]# redis-cli -h 192.168.247.133 -p 6379 -c
192.168.247.133:6379> keys *
(empty list or set)

再次添加，成功

[root@c src]# redis-cli --cluster add-node 192.168.247.133:6379 192.168.247.135:6379
[root@c src]# redis-cli --cluster check 192.168.247.135:6379		//检查
Could not connect to Redis at 192.168.247.155:6379: No route to host	
192.168.247.135:6379 (4f4175a5...) -> 1 keys | 5461 slots | 0 slaves.
192.168.247.160:6379 (0d257e1d...) -> 1 keys | 5461 slots | 1 slaves.
192.168.247.133:6379 (d5274498...) -> 0 keys | 0 slots | 0 slaves.		//添加成功
192.168.247.134:6379 (c802859b...) -> 2 keys | 5462 slots | 1 slaves.
[OK] 4 keys in 4 masters.

7.4 现在是四主二从，将新加入的133master改为slave，就给现在没有后备军的135master

发现改不了，删掉133节点，重新加入
redis-cli --cluster add-node 新节点IP：端口号 集群内任意节点IP：端口号 --cluster-slave --cluster-master-id 4f4175a5bd8245f9d708a90b97f6b20b27851ed9 //这个ID就是135master的ID

[root@c src]# redis-cli --cluster add-node 192.168.247.133:6379 192.168.247.135:6379 --cluster-slave --cluster-master-id 4f4175a5bd8245f9d708a90b97f6b20b27851ed9	//这个ID就是135master的ID
//
[root@c src]# redis-cli --cluster check 192.168.247.135:6379		//查看验证
Could not connect to Redis at 192.168.247.155:6379: No route to host
192.168.247.135:6379 (4f4175a5...) -> 1 keys | 5461 slots | 1 slaves.
192.168.247.160:6379 (0d257e1d...) -> 1 keys | 5461 slots | 1 slaves.
192.168.247.134:6379 (c802859b...) -> 2 keys | 5462 slots | 1 slaves.
[OK] 4 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.247.135:6379)
M: 4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 766fe7b89541d199b02c4074ec2aeeb317eb3ba5 192.168.247.133:6379
   slots: (0 slots) slave
   replicates 4f4175a5bd8245f9d708a90b97f6b20b27851ed9
M: 0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379
   slots: (0 slots) slave
   replicates c802859b1ebdfca96fcbe0ec48609cc8dd171dd2
S: af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379
   slots: (0 slots) slave
   replicates 0d257e1d225c2e03bc54dbf128112ed8f144be82
M: c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

备注：后面找到了将新加入的节点（新加入的节点默认为没有哈希槽的主节点）更改身份变为slave的命令

• 连接到要修改为slave的数据库
• cluster replicate 指定主节点

root@master1 ~]# redis-cli -h 192.168.100.48 -p 6379
192.168.100.48:6379> cluster replicate 99521b7fd126b694bcb9a22ffa5a490f31f66543
OK
192.168.100.48:6379> cluster nodes
a233a23541f431107fed79908318394d8bb30b51 192.168.100.43:6379@16379 master - 0 1584913315000 3 connected 12288-16383
7f112f82bcf28a5d0627ea81b48cb76f4ea8605d 192.168.100.41:6379@16379 master - 0 1584913318000 1 connected 1365-5460
df195a34a91d756157a0fda7c71e99d5bd8fad09 192.168.100.42:6379@16379 master - 0 1584913317762 2 connected 6827-10922
fa12cd664290a02f3df70a8f5a5e20950e840393 192.168.100.48:6379@16379 myself,slave 99521b7fd126b694bcb9a22ffa5a490f31f66543 0 1584913311000 0 connected
435dc9530cf424aa9c75c40904575109b865c522 192.168.100.46:6379@16379 slave df195a34a91d756157a0fda7c71e99d5bd8fad09 0 1584913314735 2 connected
0d24631322a5f92af2b32f5183ef2988851a9404 192.168.100.45:6379@16379 slave 7f112f82bcf28a5d0627ea81b48cb76f4ea8605d 0 1584913315744 1 connected
99521b7fd126b694bcb9a22ffa5a490f31f66543 192.168.100.47:6379@16379 master - 0 1584913317000 7 connected 0-1364 5461-6826 10923-12287
25f79699d0fdaa22275f4ab479d134b677159aa5 192.168.100.44:6379@16379 slave a233a23541f431107fed79908318394d8bb30b51 0 1584913318771 3 connected

1) CLUSTER <subcommand> arg arg ...arg. Subcommands are:
集群<子命令> arg arg…arg。子命令是:
2) ADDSLOTS <slot> [slot ...] -- Assign slots to current node.
addslot <slot> [slot…]——将插槽分配给当前节点。
3) BUMPEPOCH -- Advance the cluster config epoch.
3) BUMPEPOCH——推进集群配置历元。
4) COUNT-failure-reports <node-id> -- Return number of failure reports for <node-id>.
4) COUNT-failure-reports <node-id>—返回<node-id>的故障报告数量。
5) COUNTKEYSINSLOT <slot> - Return the number of keys in <slot>.
5) COUNTKEYSINSLOT <slot>—返回<slot>中的键数。
6) DELSLOTS <slot> [slot ...] -- Delete slots information from current node.
delslot <slot> [slot…——从当前节点删除插槽信息。
7) FAILOVER [force|takeover] -- Promote current replica node to being a master.
7)故障转移[强制|接管]——将当前副本节点提升为一个主节点。
8) FORGET <node-id> -- Remove a node from the cluster.
8)忘记<node-id>——从集群中删除一个节点。
9) GETKEYSINSLOT <slot> <count> -- Return key names stored by current node in a slot.
9) GETKEYSINSLOT <slot> <count>—返回当前节点在一个slot中存储的密钥名。
10) FLUSHSLOTS -- Delete current node own slots information.
10) FLUSHSLOTS——删除当前节点拥有的插槽信息。
11) INFO - Return onformation about the cluster.
11)信息-返回集群的信息。
12) KEYSLOT <key> -- Return the hash slot for <key>.
12) KEYSLOT <key>——返回<key>的哈希槽。
13) MEET <ip> <port> [bus-port] -- Connect nodes into a working cluster.
13)满足<ip> <port> [bus-port]——将节点连接到一个工作集群中。
14) MYID -- Return the node id.
14) MYID——返回节点id。
15)NODES -- Return cluster configuration seen by node. Output format:
节点——返回节点看到的集群配置。输出格式:
16) <id> <ip:port> <flags> <master> <pings> <pongs> <epoch> <link> <slot> ...<slot>
16) <id> <ip:port> <flags> <master> <pings> <pongs> <epoch> <link> <slot>…<槽>
17) REPLICATE <node-id> -- Configure current node as replica to <node-id>.
17)复制<node-id>——将当前节点配置为复制到<node-id>。
18) RESET [hard|soft] -- Reset current node (default: soft).
18)复位[硬|软]——复位当前节点(默认:软)。
19) SET-config-epoch <epoch> - Set config epoch of current node.
设置配置历元<epoch> -设置当前节点的配置历元。
20) SETSLOT <slot> (importing|migrating|stable|node <node-id>) -- Set slot state.
20) SETSLOT <slot>(导入|迁移|稳定|节点<node-id>)——设置slot状态。
21) REPLICAS <node-id> -- Return <node-id> replicas.
21)副本<node-id>——返回<node-id>副本。
22) SLOTS -- Return information about slots range mappings. Each range is made of:
slot——返回关于slot范围映射的信息。每个范围包括:
23) start, end, master and replicas IP addresses, ports and ids
23)启动、结束、主机和副本的IP地址、端口和id

7.5 将155失效节点删掉

也可以使用这个命令‘
redis-cli --cluster del-node 192.168.247.135:6379 28669db5a8328297b270018e4ef84618a2a0da9a
redis-cli --cluster del-node 集群任意节点IP：端口号 要删除的节点ID

192.168.247.135:6379> cluster forget 28669db5a8328297b270018e4ef84618a2a0da9a 		//指定要删除的节点ID
OK
192.168.247.135:6379> cluster nodes
0d257e1d225c2e03bc54dbf128112ed8f144be82 192.168.247.160:6379@16379 master - 0 1584870460548 3 connected 10923-16383
d52744987738836754b7981eb995313badbc1266 192.168.247.133:6379@16379 master - 0 1584870461556 0 connected
187619d3e5f3e6164100a3c672d2972bada89c22 192.168.247.136:6379@16379 slave c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 0 1584870460000 6 connected
af726bd42e39a0f67f1ea11d76b63c9d36e2075b 192.168.247.20:6379@16379 slave 0d257e1d225c2e03bc54dbf128112ed8f144be82 0 1584870458000 4 connected
c802859b1ebdfca96fcbe0ec48609cc8dd171dd2 192.168.247.134:6379@16379 master - 0 1584870458530 2 connected 5461-10922
4f4175a5bd8245f9d708a90b97f6b20b27851ed9 192.168.247.135:6379@16379 myself,master - 0 1584870459000 7 connected 0-5460

7.6 添加主节点

为了方便，把133从集群中剔除掉，然后再重新加入

192.168.247.135:6379> cluster forget 766fe7b89541d199b02c4074ec2aeeb317eb3ba5	//133节点ID
OK

[root@c src]# redis-cli --cluster add-node 192.168.247.133:6379 192.168.247.135:6379
[root@c src]# redis-cli --cluster check 192.168.247.135:6379
M: 051db1fd7fe2a7bb23d2ab48360cb26ad979bcd8 192.168.247.133:6379
   slots: (0 slots) master

7.6.1 重新分配集群的哈希槽，使新增节点可用

[root@c src]# redis-cli --cluster reshard 192.168.247.133:6379			//重新分配节点哈希槽
How many slots do you want to move (from 1 to 16384)? 1000			//你想要移动多少个哈希槽
What is the receiving node ID? 051db1fd7fe2a7bb23d2ab48360cb26ad979bcd8	//输入目的节点ID
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: all		//输入all，每个节点分摊
Do you want to proceed with the proposed reshard plan (yes/no)? yes	//输入yes，开始移动

7.6.2 查看分配完毕后的状态

[root@c src]# redis-cli --cluster check 192.168.247.133:6379
M: 051db1fd7fe2a7bb23d2ab48360cb26ad979bcd8 192.168.247.133:6379
   slots:[0-332],[5461-5794],[10923-11255] (1000 slots) master

分配成功

7.7 恢复数据

只需要将aof文件移动到Data dir目录下 : /var/lib/redis/6379
然后重启redis即可恢复
恢复的时候不可以写入数据，以免产生紊乱

7.8 其他扩展命令

检测各个节点槽的均衡性

redis-cli --cluster rebalance 192.168.100.41:6379 检测各个节点槽的均衡性

[root@master1 ~]# redis-cli --cluster rebalance 192.168.100.41:6379
>>> Performing Cluster Check (using node 192.168.100.41:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
*** No rebalancing needed! All nodes are within the 2.00% threshold.
//可以看出，节点负责的槽数据差异在2%以内，因此槽分配均衡

平衡各节点槽数量

[root@master1 ~]# redis-cli --cluster rebalance --cluster-threshold 1 192.168.100.41:6379
>>> Performing Cluster Check (using node 192.168.100.41:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
*** No rebalancing needed! All nodes are within the 1.00% threshold.