Redis 搭建集群高可用


Redis5.0 前采用 redis-trib 进行集群的创建和管理,需要 ruby 支持

Redis5.0 可以直接使用 Redis-cli 进行集群的创建和管理

Redis 安装启动

#第一步:安装 C 语言需要的 GCC 环境
yum install -y gcc-c++ 
yum install -y wget
#第二步:下载并解压缩 Redis 源码压缩包
wget http://download.redis.io/releases/redis-5.0.5.tar.gz 
tar -zxf redis-5.0.5.tar.gz
#第三步:编译 Redis 源码,进入 redis-5.0.5 目录,执行编译命令
cd redis-5.0.5/src 
make
#第四步:安装 Redis,需要通过 PREFIX 指定安装路径
mkdir /usr/redis -p 
make install PREFIX=/usr/redis
#复制conf
cp redis.conf /usr/redis/bin/
cp sentinel.conf /usr/redis/bin/
#修改redis.conf
vim redis.conf
#注释掉则不限于本机器访问
#bind 127.0.0.1
#no 允许外界访问
protected-mode no
#yes 守护进程后台启动
daemonize yes
#占用窗口启动 --不可以被远程访问
./redis-server
#关闭
control + c
#不占用窗口启动 --不可以被远程访问
./redis-server & 
#守护进程后台启动 --可以被远程访问
 ./redis-server redis.conf
#守护进程后台启动关闭
./redis-cli shutdown  
#查看进程
ps -ef |grep redis
root       7891   1977  0 17:14 pts/1    00:00:04 ./redis-server *:6379
#关闭进程
kill -9 7891
#客户端启动
./redis-cli -h 127.0.0.1 -p 6379
# 关闭centos的防火墙
#systemctl stop firewalld
# 设置centos防火墙不开机自启动
#systemctl disable firewalld.service

Redis 集群搭建环境

采用 3 组 redis 搭建每组包含一主二从

redis-m1-7010、redis-s11-7011、redis-s12-7012

redis-m2-7020、redis-s21-7021、redis-s22-7022

redis-m3-7030、redis-s31-7031、redis-s32-7032

Redis 集群搭建实现

1、复制一份 redis

注意如果不是新安装的 redis 要清空所有的数据

cp -r redis redis-cluster/redis-m1-7010
cd redis-cluster/redis-m1-7010
cd bin 
vim redis.conf 
#修改内容
port 7010
cluster-enabled yes
#下面这些应该已经该过了,可以再确认下
#注释掉则不限于本机器访问
#bind 127.0.0.1
#no 允许外界访问
protected-mode no
#yes 守护进程后台启动
daemonize yes
2、复制八份 redis 如下
➜  redis-cluster cp -r redis-m1-7010 redis-s11-7011       
➜  redis-cluster cp -r redis-m1-7010 redis-s12-7012 
➜  redis-cluster cp -r redis-m1-7010 redis-s21-7021 
➜  redis-cluster cp -r redis-m1-7010 redis-s22-7022 
➜  redis-cluster cp -r redis-m1-7010 redis-s31-7031 
➜  redis-cluster cp -r redis-m1-7010 redis-s32-7032 
➜  redis-cluster cp -r redis-m1-7010 redis-m2-7020  
➜  redis-cluster cp -r redis-m1-7010 redis-m3-7030 
➜  redis-cluster ls
redis-m1-7010  redis-m3-7030   redis-s12-7012  redis-s22-7022  redis-s32-7032
redis-m2-7020  redis-s11-7011  redis-s21-7021  redis-s31-7031
3、分别修改八份 redis 的端口号
vim redis-m2-7020/bin/redis.conf
port 7020
vim redis-m3-7030/bin/redis.conf
port 7030
vim redis-s11-7011/bin/redis.conf
port 7011
vim redis-s12-7012/bin/redis.conf
port 7012
vim redis-s21-7021/bin/redis.conf
port 7021
vim redis-s22-7022/bin/redis.conf
port 7022
vim redis-s31-7031/bin/redis.conf
port 7031
vim redis-s32-7032/bin/redis.conf
port 7032
4、启动九个 redis,通过 shell 命令
vim start.sh
#添加如下内容
cd redis-m1-7010/bin
./redis-server redis.conf
cd ../../
cd redis-m2-7020/bin
./redis-server redis.conf
cd ../../
cd redis-m3-7030/bin
./redis-server redis.conf
cd ../../
cd redis-s11-7011/bin/
./redis-server redis.conf
cd ../../
cd redis-s12-7012/bin/
./redis-server redis.conf
cd ../../
cd redis-s21-7021/bin/
./redis-server redis.conf
cd ../../
cd redis-s22-7022/bin/
./redis-server redis.conf
cd ../../
cd redis-s31-7031/bin/
./redis-server redis.conf
cd ../../
cd redis-s32-7032/bin/
./redis-server redis.conf
cd ../../
#赋予写和执行权限
chmod u+x start.sh
#启动
./start.sh
#查看是否启动成功
➜  redis-cluster ps - ef |grep redis 
root       8581      1  0 16:44 ?        00:00:00 ./redis-server *:7010 [cluster]
root       8583      1  0 16:44 ?        00:00:00 ./redis-server *:7020 [cluster]
root       8585      1  0 16:44 ?        00:00:00 ./redis-server *:7030 [cluster]
root       8593      1  0 16:44 ?        00:00:00 ./redis-server *:7011 [cluster]
root       8601      1  0 16:44 ?        00:00:00 ./redis-server *:7012 [cluster]
root       8606      1  0 16:44 ?        00:00:00 ./redis-server *:7021 [cluster]
root       8611      1  0 16:44 ?        00:00:00 ./redis-server *:7022 [cluster]
root       8613      1  0 16:44 ?        00:00:00 ./redis-server *:7031 [cluster]
root       8615      1  0 16:44 ?        00:00:00 ./redis-server *:7032 [cluster]
5、创建 Redis 集群(创建时 Redis 里不要有数据)
cd redis-m1-7010/bin
#查看下自己主机的IP
ifconfig
172.16.94.13
#配置主从,前面三台是主服务器、后面六台是从服务器、指定每台主机配2台从服务器,
#指定密码 没有则不需要添加密码 -a password
./redis-cli --cluster create 
172.16.94.13:7010 172.16.94.13:7020 172.16.94.13:7030 
172.16.94.13:7011 172.16.94.13:7012
172.16.94.13:7021 172.16.94.13:7022
172.16.94.13:7031 172.16.94.13:7032
 --cluster-replicas 2 

因为不能换行,不换行的命令放在了下面

./redis-cli --cluster create 172.16.94.13:7010 172.16.94.13:7020 172.16.94.13:7030 172.16.94.13:7011 172.16.94.13:7012 172.16.94.13:7021 172.16.94.13:7022 172.16.94.13:7031 172.16.94.13:7032 --cluster-replicas 2
6、具体执行集群搭建命令
➜  bin ./redis-cli --cluster create 172.16.94.13:7010 172.16.94.13:7020 172.16.94.13:7030 172.16.94.13:7011 172.16.94.13:7012 172.16.94.13:7021 172.16.94.13:7022 172.16.94.13:7031 172.16.94.13:7032 --cluster-replicas 2
>>> Performing hash slots allocation on 9 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.16.94.13:7012 to 172.16.94.13:7010
Adding replica 172.16.94.13:7021 to 172.16.94.13:7010
Adding replica 172.16.94.13:7022 to 172.16.94.13:7020
Adding replica 172.16.94.13:7031 to 172.16.94.13:7020
Adding replica 172.16.94.13:7032 to 172.16.94.13:7030
Adding replica 172.16.94.13:7011 to 172.16.94.13:7030
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: cc4de946027537864515c743c4a088524b3bde21 172.16.94.13:7010
   slots:[0-5460],[5798],[6751],[7141],[7198],[7671],[7967],[8209],[8319],[8545],[8949],[9691],[9842],[11389],[13102],[13872],[14646] (5461 slots) master
M: 47e8bb51bc3627e11871bc6d54fbd7d9b3ed09a4 172.16.94.13:7020
   slots:[86],[93],[392],[560],[845],[1463],[1637],[1780],[2169],[3083],[4354],[4773],[5360],[5449],[5461-10922],[11389],[13102],[13872],[14646] (5462 slots) master
M: a076fba9d6853ea8f0b6c8ace5fce62be25b6029 172.16.94.13:7030
   slots:[86],[93],[392],[560],[845],[1463],[1637],[1780],[2169],[3083],[4354],[4773],[5360],[5449],[5798],[6751],[7141],[7198],[7671],[7967],[8209],[8319],[8545],[8949],[9691],[9842],[10923-16383] (5461 slots) master
S: c80e948557f9592bd30e0bd6511f198da878b763 172.16.94.13:7011
   replicates 47e8bb51bc3627e11871bc6d54fbd7d9b3ed09a4
S: bfcc7f60691c74068af06a92a3a986403a286ba5 172.16.94.13:7012
   replicates cc4de946027537864515c743c4a088524b3bde21
S: 91d0d16dbc2267c8bf2d0649dba58d6d64b73c54 172.16.94.13:7021
   replicates cc4de946027537864515c743c4a088524b3bde21
S: b2d083413eeec32b9cbabc0dcc358dca733f0453 172.16.94.13:7022
   replicates a076fba9d6853ea8f0b6c8ace5fce62be25b6029
S: 1c8f25e17a4a1bda0d223a3d360ccb479da1ef8d 172.16.94.13:7031
   replicates 47e8bb51bc3627e11871bc6d54fbd7d9b3ed09a4
S: 6b2aab6bf3bea6254e532206724c30190c05426e 172.16.94.13:7032
   replicates a076fba9d6853ea8f0b6c8ace5fce62be25b6029
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.16.94.13:7010)
S: cc4de946027537864515c743c4a088524b3bde21 172.16.94.13:7010
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: b2d083413eeec32b9cbabc0dcc358dca733f0453 172.16.94.13:7022
   slots: (0 slots) slave
   replicates a076fba9d6853ea8f0b6c8ace5fce62be25b6029
S: 1c8f25e17a4a1bda0d223a3d360ccb479da1ef8d 172.16.94.13:7031
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
M: 6b2aab6bf3bea6254e532206724c30190c05426e 172.16.94.13:7032
   slots:[86],[93],[392],[560],[845],[1463],[1637],[1780],[2169],[3083],[4354],[4773],[5360],[5449],[5798],[6751],[7141],[7198],[7671],[7967],[8209],[8319],[8545],[8949],[9691],[9842],[11389],[13102],[13872],[14646] (30 slots) master
   7 additional replica(s)
S: 91d0d16dbc2267c8bf2d0649dba58d6d64b73c54 172.16.94.13:7021
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: a076fba9d6853ea8f0b6c8ace5fce62be25b6029 172.16.94.13:7030
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: bfcc7f60691c74068af06a92a3a986403a286ba5 172.16.94.13:7012
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: 47e8bb51bc3627e11871bc6d54fbd7d9b3ed09a4 172.16.94.13:7020
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: c80e948557f9592bd30e0bd6511f198da878b763 172.16.94.13:7011
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.

可以看到 出现了一个错误 [ERR] Not all 16384 slots are covered by nodes

执行下面的操作进行修复

#修复问题
#若有密码要在后面添加  -a password
./redis-cli --cluster fix 172.16.94.13:7010 
#重新分片
#./redis-cli --cluster reshard 172.16.94.13:7010
#再次检查集群
./redis-cli --cluster check 172.16.94.13:7010

#执行效果
➜  bin ./redis-cli --cluster check 172.16.94.13:7010
172.16.94.13:7032 (6b2aab6b...) -> 0 keys | 16384 slots | 7 slaves.
[OK] 0 keys in 1 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 172.16.94.13:7010)
S: cc4de946027537864515c743c4a088524b3bde21 172.16.94.13:7010
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: b2d083413eeec32b9cbabc0dcc358dca733f0453 172.16.94.13:7022
   slots: (0 slots) slave
   replicates a076fba9d6853ea8f0b6c8ace5fce62be25b6029
S: 1c8f25e17a4a1bda0d223a3d360ccb479da1ef8d 172.16.94.13:7031
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
M: 6b2aab6bf3bea6254e532206724c30190c05426e 172.16.94.13:7032
   slots:[0-16383] (16384 slots) master
   7 additional replica(s)
S: 91d0d16dbc2267c8bf2d0649dba58d6d64b73c54 172.16.94.13:7021
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: a076fba9d6853ea8f0b6c8ace5fce62be25b6029 172.16.94.13:7030
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: bfcc7f60691c74068af06a92a3a986403a286ba5 172.16.94.13:7012
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: 47e8bb51bc3627e11871bc6d54fbd7d9b3ed09a4 172.16.94.13:7020
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
S: c80e948557f9592bd30e0bd6511f198da878b763 172.16.94.13:7011
   slots: (0 slots) slave
   replicates 6b2aab6bf3bea6254e532206724c30190c05426e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

坑爹!我要的是三主六从,不是一主八从

可能是我创建集群那个命令的时候,yes 输入的慢了一些

那么重新再来,正常可以搭建成功的。。。

Redis 集群的删除

1、关闭所有的 redis 服务
➜  bin ps -ef | grep redis
root       8581      1  0 17:18 ?        00:00:07 ./redis-server *:7010 [cluster]
root       8583      1  0 17:18 ?        00:00:07 ./redis-server *:7020 [cluster]
root       8585      1  0 17:18 ?        00:00:07 ./redis-server *:7030 [cluster]
root       8593      1  0 17:18 ?        00:00:07 ./redis-server *:7011 [cluster]
root       8601      1  0 17:18 ?        00:00:07 ./redis-server *:7012 [cluster]
root       8606      1  0 17:18 ?        00:00:07 ./redis-server *:7021 [cluster]
root       8611      1  0 17:18 ?        00:00:07 ./redis-server *:7022 [cluster]
root       8613      1  0 17:18 ?        00:00:07 ./redis-server *:7031 [cluster]
root       8615      1  0 17:18 ?        00:00:20 ./redis-server *:7032 [cluster]
➜  bin kill -9 8581
➜  bin kill -9 8583
➜  bin kill -9 8585
➜  bin kill -9 8593
➜  bin kill -9 8601
➜  bin kill -9 8606
➜  bin kill -9 8611
➜  bin kill -9 8613
➜  bin kill -9 8615
➜  bin ps -ef | grep redis
➜  bin 
2、删除所有的集群和数据文件
#删除每个redis的bin下面的
dump.rdb
appendonly.aof
nodes.conf

image.png

使用 shell 脚本实现

vim delete.sh
#添加如下内容
cd redis-m1-7010/bin
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
cd redis-m2-7020/bin
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
cd redis-m3-7030/bin
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
cd redis-s11-7011/bin/
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
cd redis-s12-7012/bin/
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
cd redis-s21-7021/bin/
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
cd redis-s22-7022/bin/
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
cd redis-s31-7031/bin/
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
cd redis-s32-7032/bin/
rm appendonly.aof
rm dump.rdb
rm nodes.conf
cd ../../
#赋予写和执行权限
chmod u+x delete.sh
#执行删除
sh delete.sh

Redis 集群的重新构建

启动所有的 redis
#搭建的时候已经写好了的shell脚本直接执行
sh start.sh
➜  redis-cluster sh start.sh
9186:C 07 Aug 2021 18:39:08.394 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9186:C 07 Aug 2021 18:39:08.394 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9186, just started
9186:C 07 Aug 2021 18:39:08.394 # Configuration loaded
9188:C 07 Aug 2021 18:39:08.400 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9188:C 07 Aug 2021 18:39:08.400 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9188, just started
9188:C 07 Aug 2021 18:39:08.400 # Configuration loaded
9190:C 07 Aug 2021 18:39:08.406 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9190:C 07 Aug 2021 18:39:08.406 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9190, just started
9190:C 07 Aug 2021 18:39:08.406 # Configuration loaded
9195:C 07 Aug 2021 18:39:08.411 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9195:C 07 Aug 2021 18:39:08.411 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9195, just started
9195:C 07 Aug 2021 18:39:08.411 # Configuration loaded
9200:C 07 Aug 2021 18:39:08.417 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9200:C 07 Aug 2021 18:39:08.417 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9200, just started
9200:C 07 Aug 2021 18:39:08.417 # Configuration loaded
9202:C 07 Aug 2021 18:39:08.423 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9202:C 07 Aug 2021 18:39:08.423 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9202, just started
9202:C 07 Aug 2021 18:39:08.423 # Configuration loaded
9210:C 07 Aug 2021 18:39:08.433 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9210:C 07 Aug 2021 18:39:08.433 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9210, just started
9210:C 07 Aug 2021 18:39:08.433 # Configuration loaded
9215:C 07 Aug 2021 18:39:08.438 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9215:C 07 Aug 2021 18:39:08.438 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9215, just started
9215:C 07 Aug 2021 18:39:08.438 # Configuration loaded
9220:C 07 Aug 2021 18:39:08.442 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
9220:C 07 Aug 2021 18:39:08.442 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=9220, just started
9220:C 07 Aug 2021 18:39:08.442 # Configuration loaded

创建集群命令

cd /usr/redis-cluster/redis-m1-7010/bin
➜  bin ./redis-cli --cluster create 172.16.94.13:7010 172.16.94.13:7020 172.16.94.13:7030 172.16.94.13:7011 172.16.94.13:7012 172.16.94.13:7021 172.16.94.13:7022 172.16.94.13:7031 172.16.94.13:7032 --cluster-replicas 2
>>> Performing hash slots allocation on 9 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.16.94.13:7012 to 172.16.94.13:7010
Adding replica 172.16.94.13:7021 to 172.16.94.13:7010
Adding replica 172.16.94.13:7022 to 172.16.94.13:7020
Adding replica 172.16.94.13:7031 to 172.16.94.13:7020
Adding replica 172.16.94.13:7032 to 172.16.94.13:7030
Adding replica 172.16.94.13:7011 to 172.16.94.13:7030
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010
   slots:[0-5460] (5461 slots) master
M: f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020
   slots:[5461-10922] (5462 slots) master
M: b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030
   slots:[10923-16383] (5461 slots) master
S: 49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011
   replicates f2b1c83ac812ef667019d50d70c918beaa94eb8c
S: e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012
   replicates b9e24bb366273e16022b37262931f2c887c7b391
S: a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021
   replicates f2b1c83ac812ef667019d50d70c918beaa94eb8c
S: 260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022
   replicates ee286e7fb5fe573c46486318ce096ab23ae34b1f
S: a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031
   replicates ee286e7fb5fe573c46486318ce096ab23ae34b1f
S: 5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032
   replicates b9e24bb366273e16022b37262931f2c887c7b391
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...............
>>> Performing Cluster Check (using node 172.16.94.13:7010)
M: ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010
   slots:[0-5460] (5461 slots) master
   2 additional replica(s)
S: e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012
   slots: (0 slots) slave
   replicates b9e24bb366273e16022b37262931f2c887c7b391
S: 260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022
   slots: (0 slots) slave
   replicates ee286e7fb5fe573c46486318ce096ab23ae34b1f
M: b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030
   slots:[10923-16383] (5461 slots) master
   2 additional replica(s)
S: a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021
   slots: (0 slots) slave
   replicates f2b1c83ac812ef667019d50d70c918beaa94eb8c
S: 49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011
   slots: (0 slots) slave
   replicates f2b1c83ac812ef667019d50d70c918beaa94eb8c
S: a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031
   slots: (0 slots) slave
   replicates ee286e7fb5fe573c46486318ce096ab23ae34b1f
M: f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020
   slots:[5461-10922] (5462 slots) master
   2 additional replica(s)
S: 5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032
   slots: (0 slots) slave
   replicates b9e24bb366273e16022b37262931f2c887c7b391
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

可以看到三主六从创建 成功 slot 成功分配均匀连续完全

M1: ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010
   slots:[0-5460] (5461 slots) master
M2: f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020
   slots:[5461-10922] (5462 slots) master
M3: b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030
   slots:[10923-16383] (5461 slots) master

Redis 客户端连接集群查看

查看集群的主从信息

命令客户端连接集群测试

注意:-c 表示是以 redis 集群方式进行连接

注意 防火墙要打开端口号和端口号 +10000 的端口号

➜  bin ./redis-cli -h 127.0.0.1 -p 7010 -c
127.0.0.1:7010> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:3
cluster_current_epoch:9
cluster_my_epoch:1
cluster_stats_messages_ping_sent:715
cluster_stats_messages_pong_sent:721
cluster_stats_messages_sent:1436
cluster_stats_messages_ping_received:713
cluster_stats_messages_pong_received:715
cluster_stats_messages_meet_received:8
cluster_stats_messages_received:1436
127.0.0.1:7010> cluster nodes
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628333491000 5 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 slave ee286e7fb5fe573c46486318ce096ab23ae34b1f 0 1628333493066 7 connected
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 myself,master - 0 1628333490000 1 connected 0-5460
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628333494078 3 connected 10923-16383
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 slave f2b1c83ac812ef667019d50d70c918beaa94eb8c 0 1628333493000 6 connected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 slave f2b1c83ac812ef667019d50d70c918beaa94eb8c 0 1628333492000 4 connected
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave ee286e7fb5fe573c46486318ce096ab23ae34b1f 0 1628333492048 8 connected
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 master - 0 1628333492000 2 connected 5461-10922
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628333490000 9 connected
127.0.0.1:7010> 

可以看到 master 和 slave 的端口没错,但是其对于关系和我们预期的不一致

#master
e2 172.16.94.13:7010@17010 master
f2b 172.16.94.13:7020@17020 master
b9e 172.16.94.13:7030@17030 master
#slave
e24 172.16.94.13:7012@17012 slave 对应的 master b9e  7030
260 172.16.94.13:7022@17022 slave 对应的 maste ee2 7010
a96 172.16.94.13:7021@17021 slave f2b 7020
49d 172.16.94.13:7011@17011 slave f2b 7020
a05 172.16.94.13:7031@17031 slave ee2 7010
568 172.16.94.13:7032@17032 slave b9e 7030
测试集群的分片信息
./redis-cli -h 127.0.0.1 -p 7010 -c
127.0.0.1:7010> set name:001 lane
OK
127.0.0.1:7010> get name:001
"lane"
127.0.0.1:7010> set name:002 lane2
-> Redirected to slot [8545] located at 172.16.94.13:7020
OK
172.16.94.13:7020> get name:002
"lane2"
172.16.94.13:7020> set name:003 lane3
-> Redirected to slot [12608] located at 172.16.94.13:7030
OK
172.16.94.13:7030> get name:003
"lane3"
172.16.94.13:7030> set name:004 lane4
-> Redirected to slot [423] located at 172.16.94.13:7010
OK
172.16.94.13:7010> get name:004
"lane4"
172.16.94.13:7010> get name:002
-> Redirected to slot [8545] located at 172.16.94.13:7020
"lane2"
172.16.94.13:7010> cluster keyslot name:002
(integer) 8545

可以看到自动进行按槽分配 redis 去存储

再查看下主从中的从 redis 服务器

➜  bin ./redis-cli -p 7011
127.0.0.1:7011> keys *
1) "name:002"
127.0.0.1:7011> set aa bb
(error) MOVED 1180 172.16.94.13:7010

可以看到 7011 成功的存储了其主服务器 7020 的数据

Redis 集群挂掉

当无法完成投票选举,既是半数以上的主机器挂掉,集群就会挂掉了

先挂掉一台主服务器

➜  bin ./redis-cli -p 7010
127.0.0.1:7010> shutdown
not connected> quit
➜  bin ps -ef |grep redis
root       9189      1  0 18:39 ?        00:00:03 ./redis-server *:7020 [cluster]
root       9194      1  0 18:39 ?        00:00:03 ./redis-server *:7030 [cluster]
root       9196      1  0 18:39 ?        00:00:03 ./redis-server *:7011 [cluster]
root       9201      1  0 18:39 ?        00:00:03 ./redis-server *:7012 [cluster]
root       9206      1  0 18:39 ?        00:00:03 ./redis-server *:7021 [cluster]
root       9214      1  0 18:39 ?        00:00:03 ./redis-server *:7022 [cluster]
root       9219      1  0 18:39 ?        00:00:03 ./redis-server *:7031 [cluster]
root       9221      1  0 18:39 ?        00:00:03 ./redis-server *:7032 [cluster]
root       9256   8840  0 18:50 pts/0    00:00:00 ./redis-cli -h 127.0.0.1 -p 7010 -c

再次查看集群

➜  bin ./redis-cli -h 127.0.0.1 -p 7020 -c
127.0.0.1:7020> get name:001
(error) CLUSTERDOWN The cluster is down
127.0.0.1:7020> get name:002
"lane2"
127.0.0.1:7020> get name:003
-> Redirected to slot [12608] located at 172.16.94.13:7030
"lane3"
172.16.94.13:7030> get name:004
-> Redirected to slot [423] located at 172.16.94.13:7022
"lane4"
172.16.94.13:7022> get name:001
"lane"
172.16.94.13:7022> 

发现 7010 的一台从服务器 7022 成功上位了

再次挂掉一台主服务器 7020

➜  bin ./redis-cli -c -p 7020 
127.0.0.1:7020> shutdown
not connected> quit
➜  bin ps -ef |grep redis               
root       9194      1  0 18:39 ?        00:00:03 ./redis-server *:7030 [cluster]
root       9196      1  0 18:39 ?        00:00:03 ./redis-server *:7011 [cluster]
root       9201      1  0 18:39 ?        00:00:03 ./redis-server *:7012 [cluster]
root       9206      1  0 18:39 ?        00:00:03 ./redis-server *:7021 [cluster]
root       9214      1  0 18:39 ?        00:00:03 ./redis-server *:7022 [cluster]
root       9219      1  0 18:39 ?        00:00:03 ./redis-server *:7031 [cluster]
root       9221      1  0 18:39 ?        00:00:03 ./redis-server *:7032 [cluster]
root       9314   8840  0 19:19 pts/0    00:00:00 ./redis-cli -h 127.0.0.1 -p 7020 -c

再次查看

➜  bin ./redis-cli -h 127.0.0.1 -p 7030 -c
127.0.0.1:7030> get name:001
-> Redirected to slot [4354] located at 172.16.94.13:7022
"lane"
172.16.94.13:7022> get name:002
-> Redirected to slot [8545] located at 172.16.94.13:7011
"lane2"
172.16.94.13:7011> get name:003
-> Redirected to slot [12608] located at 172.16.94.13:7030
"lane3"

发现 7011 成功上位成功主服务器

那挂掉 7011 再看下

➜  bin ./redis-cli -c -p 7011
127.0.0.1:7011> shutdown
not connected> quit
➜  bin ps -ef |grep redis               
root       9194      1  0 18:39 ?        00:00:04 ./redis-server *:7030 [cluster]
root       9201      1  0 18:39 ?        00:00:04 ./redis-server *:7012 [cluster]
root       9206      1  0 18:39 ?        00:00:04 ./redis-server *:7021 [cluster]
root       9214      1  0 18:39 ?        00:00:04 ./redis-server *:7022 [cluster]
root       9219      1  0 18:39 ?        00:00:04 ./redis-server *:7031 [cluster]
root       9221      1  0 18:39 ?        00:00:04 ./redis-server *:7032 [cluster]
root       9328   8840  0 19:25 pts/0    00:00:00 ./redis-cli -h 127.0.0.1 -p 7030 -c
➜  bin 

可以看到

➜  bin ./redis-cli -h 127.0.0.1 -p 7030 -c
172.16.94.13:7030> get name:002
-> Redirected to slot [8545] located at 172.16.94.13:7021
"lane2"
172.16.94.13:7021> get name:001
-> Redirected to slot [4354] located at 172.16.94.13:7022
"lane"
172.16.94.13:7022> 

发现 7021 成功上位成功主服务器

a96 172.16.94.13:7021@17021 slave f2b 7020
49d 172.16.94.13:7011@17011 slave f2b 7020

现在 这一组一主二从 7020 挂掉、7011 挂掉、只剩下 7021 再给他挂掉

➜  bin ./redis-cli -c -p 7021
127.0.0.1:7021> shutdown
not connected> quit
➜  bin ps -ef |grep redis  
root       9194      1  0 18:39 ?        00:00:05 ./redis-server *:7030 [cluster]
root       9201      1  0 18:39 ?        00:00:04 ./redis-server *:7012 [cluster]
root       9214      1  0 18:39 ?        00:00:04 ./redis-server *:7022 [cluster]
root       9219      1  0 18:39 ?        00:00:05 ./redis-server *:7031 [cluster]
root       9221      1  0 18:39 ?        00:00:04 ./redis-server *:7032 [cluster]
root       9328   8840  0 19:25 pts/0    00:00:00 ./redis-cli -h 127.0.0.1 -p 7030 -c

再次查看

➜  bin ./redis-cli -h 127.0.0.1 -p 7030 -c
127.0.0.1:7030> get name:001
(error) CLUSTERDOWN The cluster is down
127.0.0.1:7030> get name:002
-> Redirected to slot [8545] located at 172.16.94.13:7032
"lane2"
172.16.94.13:7032> get name:003
-> Redirected to slot [12608] located at 172.16.94.13:7030
"lane3"
172.16.94.13:7030> get name:004
-> Redirected to slot [423] located at 172.16.94.13:7022
"lane4"
172.16.94.13:7022> get name:001
"lane"
172.16.94.13:7022> get name:002
-> Redirected to slot [8545] located at 172.16.94.13:7032
"lane2"
172.16.94.13:7032> set name:002 lane22
OK

7032 成功上位从 7030 的从服务器成功上位,发生了副本漂移

再次查看下所有节点的信息

172.16.94.13:7032> cluster nodes
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628336270886 12 connected
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 myself,master - 0 1628336268000 16 connected 5461-10922
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 master,fail - 1628335913067 1628335910000 14 disconnected
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628336269000 3 connected 10923-16383
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 master,fail - 1628335142606 1628335139000 1 disconnected
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628336269870 5 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 master - 0 1628336266000 12 connected 0-5460
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 master,fail - 1628335521452 1628335514000 2 disconnected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 master,fail - 1628335693957 1628335689000 13 disconnected
172.16.94.13:7032> 

或许这就高可用吧

再次重新启动所有挂掉的服务器

➜  redis-cluster sh start.sh 
➜  redis-cluster ./redis-cli -h 127.0.0.1 -p 7030 -c
cd /usr/redis-cluster/redis-m1-7010/bin
➜  bin ./redis-cli -h 127.0.0.1 -p 7010 -c
127.0.0.1:7010> cluster nodes
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 myself,slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628336514000 1 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 master - 0 1628336520001 12 connected 0-5460
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628336516000 16 connected
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628336517961 3 connected 10923-16383
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628336515926 5 connected
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 master - 0 1628336516000 16 connected 5461-10922
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628336519000 16 connected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628336517000 16 connected
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628336518000 12 connected
127.0.0.1:7010> 

可以看到集群成功启动

master 分别为 7022、7032、7030

7032 有三台从服务器 7011、7021、7022

7030 有一台从服务器 7012

7022 有两台从服务器 7010、7031

Redis 集群的扩容

添加一个新的 redis ,注意 旧的 redis 不可以了,有数据的也不可以

添加一个 Redis
#第一步:安装 C 语言需要的 GCC 环境
yum install -y gcc-c++ 
yum install -y wget
#第二步:下载并解压缩 Redis 源码压缩包
wget http://download.redis.io/releases/redis-5.0.5.tar.gz 
tar -zxf redis-5.0.5.tar.gz
#第三步:编译 Redis 源码,进入 redis-5.0.5 目录,执行编译命令
cd redis-5.0.5/src 
#第四步:安装 Redis,需要通过 PREFIX 指定安装路径
make 
mkdir /usr/redis-cluster/redis-extend-7101
make install PREFIX=/usr/redis-cluster/redis-extend-7101
cd ../
cp redis.conf /usr/redis-cluster/redis-extend-7101/bin/
cd /usr/redis-cluster/redis-extend-7101/bin/
#修改redis.conf
vim redis.conf
port 7101
#注释掉则不限于本机器访问
#bind 127.0.0.1
#no 允许外界访问
protected-mode no
#yes 守护进程后台启动
daemonize yes
#集群
cluster-enabled yes
cd ../../
#copy三份便于测试
➜  redis-cluster cp -r redis-extend-7101 redis-extend-7100
➜  redis-cluster cp -r redis-extend-7101 redis-extend-7102
➜  redis-cluster cp -r redis-extend-7101 redis-extend-7103
#启动
cd /usr/redis-cluster/redis-extend-7101/bin/
./redis-server redis.conf
Redis 添加一个 master 节点

添加到集群当中前一个是要添加的,后一个是随便指定通信的一个集群中的机器节点,这里我指定的是一个主节点 7022

./redis-cli --cluster add-node 172.16.94.13:7101 172.16.94.13:7022

具体执行

➜  bin ./redis-cli --cluster add-node 172.16.94.13:7101 172.16.94.13:7022
>>> Adding node 172.16.94.13:7101 to cluster 172.16.94.13:7022
>>> Performing Cluster Check (using node 172.16.94.13:7022)
M: 260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022
   slots:[0-5460] (5461 slots) master
   3 additional replica(s)
S: e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012
   slots: (0 slots) slave
   replicates b9e24bb366273e16022b37262931f2c887c7b391
S: ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010
   slots: (0 slots) slave
   replicates 260ad554e37c39c39f4abeac5531f2d9189b1d3b
S: 49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011
   slots: (0 slots) slave
   replicates 5682f20105511056e3f6d90c0818ea2612597733
M: 5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032
   slots:[5461-10922] (5462 slots) master
   4 additional replica(s)
M: b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 67b124ac9bb083aaa1b021117854d97e31daff6c 172.16.94.13:7040
   slots: (0 slots) slave
   replicates 260ad554e37c39c39f4abeac5531f2d9189b1d3b
S: a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021
   slots: (0 slots) slave
   replicates 5682f20105511056e3f6d90c0818ea2612597733
S: a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031
   slots: (0 slots) slave
   replicates 260ad554e37c39c39f4abeac5531f2d9189b1d3b
S: f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020
   slots: (0 slots) slave
   replicates 5682f20105511056e3f6d90c0818ea2612597733
S: 4fb24a73d156f20a63985187b93171903325087c 172.16.94.13:7060
   slots: (0 slots) slave
   replicates 5682f20105511056e3f6d90c0818ea2612597733
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.16.94.13:7101 to make it join the cluster.
[OK] New node added correctly.

查看节点信息

➜  bin ./redis-cli -h 127.0.0.1 -p 7022
127.0.0.1:7022> cluster nodes
7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 172.16.94.13:7101@17101 master - 0 1628665528000 0 connected
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628665530000 5 connected
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628665529754 12 connected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628665528000 16 connected
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 master - 0 1628665528747 16 connected 5461-10922
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628665528000 3 connected 10923-16383
67b124ac9bb083aaa1b021117854d97e31daff6c 172.16.94.13:7040@17040 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628665530766 12 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 myself,master - 0 1628665527000 12 connected 0-5460
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628665527739 16 connected
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628665526000 12 connected
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628665527000 16 connected
4fb24a73d156f20a63985187b93171903325087c 172.16.94.13:7060@17060 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628665528644 16 connected
127.0.0.1:7022> 

可以看到 slot 的已经分配完了故而 7101 没有 slot 可以分配,需要重新分配槽才行

7032 master - 0 1628665528747 16 connected 5461-10922

7030 master - 0 1628665528000 3 connected 10923-16383

7022 myself,master - 0 1628665527000 12 connected 0-5460

Redis 集群的重新分槽
./redis-cli --cluster reshard 172.16.94.13:7101

具体执行

./redis-cli --cluster reshard 172.16.94.13:7101
#第一步:连接上集群(连接集群中任意一个可用结点都行)
#第二步:输入要分配的槽数量
#第三步:输入接收槽的结点 id
#第四步:输入源结点 id ,选择输入:all
#第五步:输入 yes 开始移动槽到目标结点 id
#迁移ing2
Moving slot 11919 from 172.16.94.13:7030 to 172.16.94.13:7101: 
Moving slot 11920 from 172.16.94.13:7030 to 172.16.94.13:7101: 
Moving slot 11921 from 172.16.94.13:7030 to 172.16.94.13:7101: 

再次查看集群

➜  bin  ./redis-cli -h 127.0.0.1 -p 7022
127.0.0.1:7022> cluster nodes
7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 172.16.94.13:7101@17101 master - 0 1628666469635 17 connected 0-998 5461-6461 10923-11921
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628666469000 5 connected
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628666470648 12 connected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628666467000 16 connected
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 master - 0 1628666467611 16 connected 6462-10922
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628666469000 3 connected 11922-16383
67b124ac9bb083aaa1b021117854d97e31daff6c 172.16.94.13:7040@17040 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628666470000 12 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 myself,master - 0 1628666464000 12 connected 999-5460
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628666468000 16 connected
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628666468000 12 connected
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628666465000 16 connected
4fb24a73d156f20a63985187b93171903325087c 172.16.94.13:7050@17050 slave 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 0 1628666468219 17 connected
127.0.0.1:7022> 

可以看到槽发生了变化,相应槽位上的数据存储的机器也发生了变化

7101 master - 0 1628666469635 17 connected 0-998 5461-6461 10923-11921

7032 master - 0 1628666467611 16 connected 6462-10922

7030 master - 0 1628666469000 3 connected 11922-16383

7022 master - 0 1628666464000 12 connected 999-5460

测试下节点插入

127.0.0.1:7022> set name:1 1
(error) MOVED 12501 172.16.94.13:7030
127.0.0.1:7022> set name:2 2
(error) MOVED 182 172.16.94.13:7101
127.0.0.1:7022> set name:3 3
OK

出现错误,是因为输入的时候忘了加上 -c

➜  bin  ./redis-cli -h 127.0.0.1 -p 7022 -c
127.0.0.1:7022> set name:1 one
-> Redirected to slot [12501] located at 172.16.94.13:7030
OK
172.16.94.13:7030> set name:2 two
-> Redirected to slot [182] located at 172.16.94.13:7101
OK
172.16.94.13:7101> set name:3 three
-> Redirected to slot [4247] located at 172.16.94.13:7022
OK
172.16.94.13:7022> get name:1
-> Redirected to slot [12501] located at 172.16.94.13:7030
"one"
172.16.94.13:7030> 
添加一个 slave 机器

把刚才复制好的 7102 修改下端口号为 7002,再重新启动

开始添加 7102 为 7101 的从机器需要指定 7101 的 ID

./redis-cli --cluster add-node 172.16.94.13:7102 172.16.94.13:7101 --cluster-slave --cluster-master-id 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff
➜  bin ./redis-cli --cluster add-node 172.16.94.13:7102 172.16.94.13:7101 --cluster-slave --cluster-master-id 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff
>>> Adding node 172.16.94.13:7102 to cluster 172.16.94.13:7101
>>> Performing Cluster Check (using node 172.16.94.13:7101)
M: 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 172.16.94.13:7101
   slots:[0-998],[5461-6461],[10923-11921] (2999 slots) master
   1 additional replica(s)
S: a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021
   slots: (0 slots) slave
   replicates 5682f20105511056e3f6d90c0818ea2612597733
S: 49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011
   slots: (0 slots) slave
   replicates 5682f20105511056e3f6d90c0818ea2612597733
S: 67b124ac9bb083aaa1b021117854d97e31daff6c 172.16.94.13:7040
   slots: (0 slots) slave
   replicates 260ad554e37c39c39f4abeac5531f2d9189b1d3b
S: e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012
   slots: (0 slots) slave
   replicates b9e24bb366273e16022b37262931f2c887c7b391
S: ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010
   slots: (0 slots) slave
   replicates 260ad554e37c39c39f4abeac5531f2d9189b1d3b
S: 4fb24a73d156f20a63985187b93171903325087c 172.16.94.13:7050
   slots: (0 slots) slave
   replicates 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff
M: 260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022
   slots:[999-5460] (4462 slots) master
   3 additional replica(s)
S: a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031
   slots: (0 slots) slave
   replicates 260ad554e37c39c39f4abeac5531f2d9189b1d3b
S: f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020
   slots: (0 slots) slave
   replicates 5682f20105511056e3f6d90c0818ea2612597733
M: b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030
   slots:[11922-16383] (4462 slots) master
   1 additional replica(s)
M: 5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032
   slots:[6462-10922] (4461 slots) master
   3 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.16.94.13:7102 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 172.16.94.13:7101.
[OK] New node added correctly.

查看现在节点信息

➜  bin ./redis-cli -h 127.0.0.1 -p 7102 -c
127.0.0.1:7102> cluster nodes
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628668824000 16 connected
34bd878c66ea8f4bfabf0bf10b9b9ff6261890ad 172.16.94.13:7102@17102 myself,slave 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 0 1628668822000 0 connected
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628668822000 12 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 master - 0 1628668824530 12 connected 999-5460
4fb24a73d156f20a63985187b93171903325087c 172.16.94.13:7050@17050 slave 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 0 1628668825946 17 connected
7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 172.16.94.13:7101@17101 master - 0 1628668825742 17 connected 0-998 5461-6461 10923-11921
67b124ac9bb083aaa1b021117854d97e31daff6c 172.16.94.13:7040@17040 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628668822000 12 connected
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628668824732 3 connected 11922-16383
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 master - 0 1628668822000 16 connected 6462-10922
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628668823000 12 connected
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628668820000 16 connected
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628668822707 3 connected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628668822000 16 connected
127.0.0.1:7102> 

测试下新添加的从机器 7102 的数据 是否和 7101 一样

➜  bin ./redis-cli -h 127.0.0.1 -p 7101 -c   
127.0.0.1:7101> get name:01
-> Redirected to slot [10136] located at 172.16.94.13:7032
(nil)
172.16.94.13:7032> get name:1
-> Redirected to slot [12501] located at 172.16.94.13:7030
"one"
172.16.94.13:7030> get name:2
-> Redirected to slot [182] located at 172.16.94.13:7101
"two"
172.16.94.13:7101> 
#可以看到7101存储了name:02 two
➜  bin ./redis-cli -h 127.0.0.1 -p 7102 -c

127.0.0.1:7102> readonly 
OK
127.0.0.1:7102> get name:2
"two"
127.0.0.1:7102> 

Redis 集群缩容

集群中删除从节点

实际只是关闭 redis 进程

./redis-cli --cluster del-node 172.16.94.13:7102 34bd878c66ea8f4bfabf0bf10b9b9ff6261890ad

具体操作,可以看到节点 7102 的 redis 被关闭了,一旦重启仍可以恢复

➜  bin ./redis-cli --cluster del-node 172.16.94.13:7102 34bd878c66ea8f4bfabf0bf10b9b9ff6261890ad
>>> Removing node 34bd878c66ea8f4bfabf0bf10b9b9ff6261890ad from cluster 172.16.94.13:7102
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
➜  bin ./redis-cli -h 127.0.0.1 -p 7102 -c                                                    
Could not connect to Redis at 127.0.0.1:7102: Connection refused
not connected> quit
➜  bin ./redis-cli -h 127.0.0.1 -p 7101 -c
127.0.0.1:7101> cluster nodes
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628668969959 16 connected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628668967000 16 connected
67b124ac9bb083aaa1b021117854d97e31daff6c 172.16.94.13:7040@17040 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628668968000 12 connected
7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 172.16.94.13:7101@17101 myself,master - 0 1628668961000 17 connected 0-998 5461-6461 10923-11921
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628668963000 3 connected
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628668967910 12 connected
4fb24a73d156f20a63985187b93171903325087c 172.16.94.13:7050@17050 slave 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 0 1628668965000 17 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 master - 0 1628668968000 12 connected 999-5460
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628668968934 12 connected
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628668967502 16 connected
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628668965858 3 connected 11922-16383
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 master - 0 1628668967000 16 connected 6462-10922
127.0.0.1:7101> quit
➜  bin ps -ef | grep redis
root       1408      1  0 13:33 ?        00:00:13 ./redis-server *:7010 [cluster]
root       1413      1  0 13:33 ?        00:00:13 ./redis-server *:7020 [cluster]
root       1418      1  0 13:33 ?        00:00:15 ./redis-server *:7030 [cluster]
root       1423      1  0 13:33 ?        00:00:13 ./redis-server *:7011 [cluster]
root       1428      1  0 13:33 ?        00:00:13 ./redis-server *:7012 [cluster]
root       1433      1  0 13:33 ?        00:00:13 ./redis-server *:7021 [cluster]
root       1438      1  0 13:33 ?        00:00:14 ./redis-server *:7022 [cluster]
root       1443      1  0 13:33 ?        00:00:13 ./redis-server *:7031 [cluster]
root       1448      1  0 13:33 ?        00:00:14 ./redis-server *:7032 [cluster]
root       1513      1  0 13:39 ?        00:00:13 ./redis-server *:7040 [cluster]
root       1582      1  0 13:56 ?        00:00:11 ./redis-server *:7050 [cluster]
root       2119      1  0 14:09 ?        00:00:09 ./redis-server *:7060 [cluster]
root       2177      1  0 14:13 ?        00:00:09 ./redis-server *:7061 [cluster]
root       7185      1  0 15:03 ?        00:00:07 ./redis-server *:7101 [cluster]
root       7352   2024  0 16:03 pts/0    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox redis
➜  bin ./redis-server redis.conf
7355:C 11 Aug 2021 16:04:01.612 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
7355:C 11 Aug 2021 16:04:01.612 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=7355, just started
7355:C 11 Aug 2021 16:04:01.612 # Configuration loaded
➜  bin ps -ef | grep redis          
root       1408      1  0 13:33 ?        00:00:13 ./redis-server *:7010 [cluster]
root       1413      1  0 13:33 ?        00:00:13 ./redis-server *:7020 [cluster]
root       1418      1  0 13:33 ?        00:00:15 ./redis-server *:7030 [cluster]
root       1423      1  0 13:33 ?        00:00:13 ./redis-server *:7011 [cluster]
root       1428      1  0 13:33 ?        00:00:13 ./redis-server *:7012 [cluster]
root       1433      1  0 13:33 ?        00:00:13 ./redis-server *:7021 [cluster]
root       1438      1  0 13:33 ?        00:00:14 ./redis-server *:7022 [cluster]
root       1443      1  0 13:33 ?        00:00:13 ./redis-server *:7031 [cluster]
root       1448      1  0 13:33 ?        00:00:14 ./redis-server *:7032 [cluster]
root       1513      1  0 13:39 ?        00:00:13 ./redis-server *:7040 [cluster]
root       1582      1  0 13:56 ?        00:00:11 ./redis-server *:7050 [cluster]
root       2119      1  0 14:09 ?        00:00:09 ./redis-server *:7060 [cluster]
root       2177      1  0 14:13 ?        00:00:09 ./redis-server *:7061 [cluster]
root       7185      1  0 15:03 ?        00:00:08 ./redis-server *:7101 [cluster]
root       7356      1  0 16:04 ?        00:00:00 ./redis-server *:7102 [cluster]
➜  bin ./redis-cli -h 127.0.0.1 -p 7101 -c
127.0.0.1:7101> cluster nodes
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628669051000 16 connected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628669052138 16 connected
67b124ac9bb083aaa1b021117854d97e31daff6c 172.16.94.13:7040@17040 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628669047000 12 connected
7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 172.16.94.13:7101@17101 myself,master - 0 1628669051000 17 connected 0-998 5461-6461 10923-11921
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628669054162 3 connected
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628669053150 12 connected
4fb24a73d156f20a63985187b93171903325087c 172.16.94.13:7061@17061 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628669053000 17 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 master - 0 1628669049000 12 connected 999-5460
34bd878c66ea8f4bfabf0bf10b9b9ff6261890ad 172.16.94.13:7102@17102 slave 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 0 1628669053000 17 connected
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628669050105 12 connected
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628669051123 16 connected
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628669052000 3 connected 11922-16383
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 master - 0 1628669050000 16 connected 6462-10922
127.0.0.1:7101> quit
➜  bin ./redis-cli -h 127.0.0.1 -p 7102 -c
127.0.0.1:7102> readonly
OK
127.0.0.1:7102> get name:2
"two"
127.0.0.1:7102> 
集群中删除主节点

必须节点 slot 空才可以

➜  bin ./redis-cli --cluster del-node 172.16.94.13:7101 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff
>>> Removing node 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff from cluster 172.16.94.13:7101
[ERR] Node 172.16.94.13:7101 is not empty! Reshard data away and try again.
直接删除该节点生成的集群配置文件 nodes.conf
➜  bin ps -ef |grep redis
root       1408      1  0 13:33 ?        00:00:14 ./redis-server *:7010 [cluster]
root       1413      1  0 13:33 ?        00:00:13 ./redis-server *:7020 [cluster]
root       1418      1  0 13:33 ?        00:00:15 ./redis-server *:7030 [cluster]
root       1423      1  0 13:33 ?        00:00:14 ./redis-server *:7011 [cluster]
root       1428      1  0 13:33 ?        00:00:13 ./redis-server *:7012 [cluster]
root       1433      1  0 13:33 ?        00:00:13 ./redis-server *:7021 [cluster]
root       1438      1  0 13:33 ?        00:00:15 ./redis-server *:7022 [cluster]
root       1443      1  0 13:33 ?        00:00:13 ./redis-server *:7031 [cluster]
root       1448      1  0 13:33 ?        00:00:15 ./redis-server *:7032 [cluster]
root       1513      1  0 13:39 ?        00:00:13 ./redis-server *:7040 [cluster]
root       1582      1  0 13:56 ?        00:00:11 ./redis-server *:7050 [cluster]
root       2119      1  0 14:09 ?        00:00:10 ./redis-server *:7060 [cluster]
root       2177      1  0 14:13 ?        00:00:10 ./redis-server *:7061 [cluster]
root       7185      1  0 15:03 ?        00:00:08 ./redis-server *:7101 [cluster]
root       7384      1  0 16:07 ?        00:00:00 ./redis-server *:7102 [cluster]
➜  bin kill -9 7384
➜  bin ls
dump.rdb  nodes.conf  redis-benchmark  redis-check-aof  redis-check-rdb  redis-cli  redis.conf  redis-sentinel  redis-server
➜  bin rm nodes.conf
➜  bin ./redis-server redis.conf                                                          
7438:C 11 Aug 2021 16:13:21.510 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
7438:C 11 Aug 2021 16:13:21.510 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=7438, just started
7438:C 11 Aug 2021 16:13:21.510 # Configuration loaded
➜  bin ./redis-cli -h 127.0.0.1 -p 7101 -c                                                
127.0.0.1:7101> cluster nodes
a963946d7e82443d334bcfaef82faa4ad0dcba20 172.16.94.13:7021@17021 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628669615000 16 connected
49dad3f2b928850e306290bcf75f7346bc2ed6fd 172.16.94.13:7011@17011 slave 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 0 1628669612498 17 connected
67b124ac9bb083aaa1b021117854d97e31daff6c 172.16.94.13:7040@17040 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628669614000 12 connected
7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 172.16.94.13:7101@17101 myself,master - 0 1628669611000 17 connected 0-998 5461-6461 10923-11921
e24cab86b8eb6312dd6f2f99c413261c814a4216 172.16.94.13:7012@17012 slave b9e24bb366273e16022b37262931f2c887c7b391 0 1628669617595 3 connected
ee286e7fb5fe573c46486318ce096ab23ae34b1f 172.16.94.13:7010@17010 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628669614000 12 connected
4fb24a73d156f20a63985187b93171903325087c 172.16.94.13:7050@17050 slave 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 0 1628669615000 17 connected
260ad554e37c39c39f4abeac5531f2d9189b1d3b 172.16.94.13:7022@17022 master - 0 1628669617000 12 connected 999-5460
34bd878c66ea8f4bfabf0bf10b9b9ff6261890ad :0@0 slave,fail,noaddr 7cedc3fccea9d4f155835fdd3ecf102306f6c4ff 1628669577414 1628669574000 17 disconnected
a05cf8a1ac31d5d25db812ad56d25e5c60893311 172.16.94.13:7031@17031 slave 260ad554e37c39c39f4abeac5531f2d9189b1d3b 0 1628669615556 12 connected
f2b1c83ac812ef667019d50d70c918beaa94eb8c 172.16.94.13:7020@17020 slave 5682f20105511056e3f6d90c0818ea2612597733 0 1628669616576 16 connected
b9e24bb366273e16022b37262931f2c887c7b391 172.16.94.13:7030@17030 master - 0 1628669615000 3 connected 11922-16383
5682f20105511056e3f6d90c0818ea2612597733 172.16.94.13:7032@17032 master - 0 1628669614000 16 connected 6462-10922
127.0.0.1:7101> quit
➜  bin ./redis-cli -h 127.0.0.1 -p 7102 -c
127.0.0.1:7102> keys *
(empty list or set)
127.0.0.1:7102> set name:101 101
(error) CLUSTERDOWN Hash slot not served
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值