redis实战--主从--sentinel--Cluster

目录

一、主从模式

1.1主从测试

1.2通过配置快照快速存储数据

1.3通过AOF日志备份数据

二、sentinel模式

2.1特点

2.2搭建

2.3启动sentinel

2.4测试sentinel

2.5 sentinel 事件说明

三、Cluster模式

3.1Cluster集群特点

3.2Cluster搭建

3.4Cluster 伸缩

3.4.1 扩容--slot分配

3.4.2 缩容--slot分配


redis集群有三种模式

1.主从,2.sentinel(哨兵),3.Cluster模式

环境:centos7 ,redis-6.0.10

一、主从模式

(一台电脑上测试)

特点:master挂了,redis从可读,影响master连接以及写操作。不会重选master。

1.1主从测试

[root@ol7-19 redis-6.0.10]# ps -aux|grep 6379
root      23328  0.5  0.1 164940  3004 pts/4    Sl+  10:58   0:16 redis-server 127.0.0.1:6379
root      25068  0.0  0.0  25044  1696 pts/7    S+   11:32   0:00 redis-cli -h 127.0.0.1 -p 6379
root      25744  0.0  0.0 112816   972 pts/0    S+   11:45   0:00 grep --color=auto 6379
[root@ol7-19 redis-6.0.10]# kill -9 23328
[root@ol7-19 redis-6.0.10]# ps -aux|grep 6379
root      25068  0.0  0.0  25044  1696 pts/7    S+   11:32   0:00 redis-cli -h 127.0.0.1 -p 6379
root      25761  0.0  0.0 112812   972 pts/0    S+   11:46   0:00 grep --color=auto 6379
[root@ol7-19 redis-6.0.10]# redis-cli
Could not connect to Redis at 127.0.0.1:6379: Connection refused
not connected> exit
[root@ol7-19 redis-6.0.10]# redis-cli -p 7000
127.0.0.1:7000> get foo
"bar"
127.0.0.1:7000>

删除master的快照文件,dump.rdb,master挂掉之后重启不会恢复数据,从库会同步,导致所有数据清空,如果使用一个快照文件。

手动kill -9停止master之前,要先使用save命令,备份数据文件。

自动退出会在退出前自动保存数据(kill 不会自动保存)

27668:M 15 Feb 2021 12:19:21.878 * Saving the final RDB snapshot before exiting.
27668:M 15 Feb 2021 12:19:21.881 * DB saved on disk
27668:M 15 Feb 2021 12:19:21.882 # Redis is now ready to exit, bye bye...

1.2通过配置快照快速存储数据

save 900 1      // 900内,有1条写入,则产生快照 
save 300 1000   // 如果300秒内有1000次写入,则产生快照
save 60 10000  // 如果60秒内有10000次写入,则产生快照
(这3个选项都屏蔽,则rdb禁用)
stop-writes-on-bgsave-error yes  // 后台备份进程出错时,主进程停不停止写入?
rdbcompression yes    // 导出的rdb文件是否压缩
Rdbchecksum   yes //  导入rbd恢复时数据时,要不要检验rdb的完整性
dbfilename dump.rdb  //导出来的rdb文件名
dir ./  //rdb的放置路径

1.3通过AOF日志备份数据

appendonly no # 是否打开 aof日志功能

appendfsync always   # 每1个命令,都立即同步到aof. 安全,速度慢
appendfsync everysec # 折衷方案,每秒写1次
appendfsync no      # 写入工作交给操作系统,由操作系统判断缓冲区大小,统一写入到aof. 同步频率低,速度快,


no-appendfsync-on-rewrite  yes: # 正在导出rdb快照的过程中,要不要停止同步aof
auto-aof-rewrite-percentage 100 #aof文件大小比起上次重写时的大小,增长率100%时,重写
auto-aof-rewrite-min-size 64mb #aof文件,至少超过64M时,重写

 

1.5 redis 生产环境启动方案(仅限只安装了一个版本的redis)

https://developer.aliyun.com/article/649990?spm=a2c6h.13813017.0.dArticle738638.316a6856d8vATl

以上配置待测试

复制redis.conf,7000.conf,7001.conf

修改7000.conf和7001.conf的端口分别不7000和7001

增加replicaof 127.0.0.1 6379

redis-server redis.conf

redis-server 7000.conf

redis-server 7001.conf

按顺序在一台服务器上分别启动不同端口的redis服务

[root@ol7-19 redis-6.0.10]# ss -antl
State       Recv-Q Send-Q                                                 Local Address:Port                                                                Peer Address:Port
LISTEN      0      128                                                        127.0.0.1:6379                                                                           *:*
LISTEN      0      128                                                                *:22                                                                             *:*
LISTEN      0      128                                                        127.0.0.1:7000                                                                           *:*
LISTEN      0      128                                                        127.0.0.1:7001                                                                           *:*
LISTEN      0      100                                                        127.0.0.1:25                                                                             *:*
LISTEN      0      128                                                             [::]:22                                                                          [::]:*
LISTEN      0      100                                                            [::1]:25                                                                          [::]:*
[root@ol7-19 redis-6.0.10]#

 12查看主从集群信息

集群命令:https://blog.csdn.net/zwjzqqb/article/details/79973909

client list:查看当前连接到redis的传话信息:id,来源ip,连接的db

 

master,会表明角色为主服务器,role:master

role命令返回当前实例在集群中的角色状态

127.0.0.1:6379> role
1) "master"
2) (integer) 3103
3) 1) 1) "127.0.0.1"
      2) "7000"
      3) "3103"
   2) 1) "127.0.0.1"
      2) "7001"
      3) "3103"
127.0.0.1:6379>
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=127.0.0.1,port=7000,state=online,offset=2067,lag=0
slave1:ip=127.0.0.1,port=7001,state=online,offset=2067,lag=0
master_replid:ea56a307c4ba2b7516a6cc53206fabeac09c8502
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2067
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:2067
127.0.0.1:6379>

slave,会表明角色为从服务器,role:slave

127.0.0.1:7001> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:1927
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:ea56a307c4ba2b7516a6cc53206fabeac09c8502
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:1927
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:1927
127.0.0.1:7001>

 

测试,主可以读写,从只能读,

[root@ol7-19 redis-6.0.10]# redis-cli -h 127.0.0.1 -p 6379
127.0.0.1:6379> del foo
(integer) 1
127.0.0.1:6379> set foo bar
OK
127.0.0.1:6379> get foo
"bar"
127.0.0.1:6379> exit
[root@ol7-19 redis-6.0.10]# redis-cli -h 127.0.0.1 -p 7000
127.0.0.1:7000> get foo
"bar"
127.0.0.1:7000> exit
[root@ol7-19 redis-6.0.10]# redis-cli -h 127.0.0.1 -p 7001
127.0.0.1:7001> get foo
"bar"
127.0.0.1:7001> del foo
(error) READONLY You can't write against a read only replica.
127.0.0.1:7001> set foo1 bar1
(error) READONLY You can't write against a read only replica.
127.0.0.1:7001>

结论,指定登陆不同的端口,在master上可以读写,在slave上写会提示:(error) READONLY You can't write against a read only replica.

不能在只读从服务器上写。

二、sentinel模式

主从模式集群不具备高可用,所以sentinel(哨兵)应运而生。

哨兵顾名思义就是监控redis集群的运行状况

2.1特点

1).建立在主从模式的基础上

2).master挂掉会在slave中选择一个为master,修改对应的配置文件,挂掉的master重启后变成slave.

3).sentinel也会挂掉,所以sentinel也要找成集群方式。

4).一个sentinel或sentinel集群可以管理多个主从Redis,多个sentinel也可以监控同一个redis

5).sentinel最好不要和Redis部署在同一台机器,不然Redis的服务器挂了以后,sentinel也挂了

2.2搭建

环境准备

rediis服务器ip端口
master172.18.28.245sentinel 26379
slave

172.18.28.244

sentinel 26379
slave172.18.28.243sentinel 26379

 

 

 

 

master修改redis.conf

sed -i 's/^bind 127.0.0.1/bind 0.0.0.0/g' redis.conf

sed -i 's/protected-mode yes/protected-mode no/g' redis.conf

sed -i 's/appendonly no/appendonly yes/g' redis.conf

sed -i "/# requirepass foobared/a requirepass 123456" redis.conf
sed -i "/# masterauth <master-password>/a masterauth 123456" redis.conf

slave修改redis.conf

sed -i 's/^bind 127.0.0.1/bind 0.0.0.0/g' redis.conf

sed -i 's/protected-mode yes/protected-mode no/g' redis.conf

sed -i 's/appendonly no/appendonly yes/g' redis.conf

sed -i "/# replicaof <masterip> <masterport>/a replicaof 172.18.28.245 6379" redis.conf
sed -i "/# requirepass foobared/a requirepass 123456" redis.conf
sed -i "/# masterauth <master-password>/a masterauth 123456" redis.conf
 

将它写成脚本config.sh,方便一个脚本多次安装:

#!/bin.bash
read -p "Setting the master?[y/n]:" input
echo $input
filename=redis.conf
filepath=/usr/local/redis-6.0.10
file=$filepath/$filename
masterIP=172.18.28.245
masterPort=6379
if [ -f "$file" ];then
echo "exist"
else
echo "not exist"
exit
fi
if [ $input = "n" ];then
IP=`ip addr|awk '{if($2~"172")print $2}'|awk -F '/' '{print $1}'`
sed -i "s/^bind 127.0.0.1/bind $IP/g" $file

sed -i 's/protected-mode yes/protected-mode no/g' $file

sed -i 's/appendonly no/appendonly yes/g' $file

sed -i "/# replicaof <masterip> <masterport>/a replicaof $masterIP $masterPort" $file
# requirepass foobared
sed -i "/# requirepass foobared/a requirepass 123456" $file
# masterauth <master-password>
sed -i "/# masterauth <master-password>/a masterauth 123456" $file

elif [ $input = "y" ];then
IP=`ip addr|awk '{if($2~"172")print $2}'|awk -F '/' '{print $1}'`
sed -i "s/^bind 127.0.0.1/bind $IP/g" $file

sed -i 's/protected-mode yes/protected-mode no/g' $file

sed -i 's/appendonly no/appendonly yes/g' $file
# requirepass foobared
sed -i "/# requirepass foobared/a requirepass 123456" $file
# masterauth <master-password>
sed -i "/# masterauth <master-password>/a masterauth 123456" $file
fi

当使用sentinel模式的时候,客户端就不要直接连接Redis,而是连接sentinel的ip和port,由sentinel来提供具体的可提供服务的Redis实现,这样当master节点挂掉以后,sentinel就会感知并将新的master节点提供给使用者。

查看sentinel.conf配置

[root@mycat redis-6.0.10]# grep -Ev "^#|^$" sentinel.conf
port 26379
daemonize no
pidfile /var/run/redis-sentinel.pid
logfile ""
dir /tmp
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
[root@mycat redis-6.0.10]#

修改为

sed -i "s/127.0.0.1/172.18.28.245/g" sentinel.conf

sed -i "/^sentinel monitor/a sentinel auth-pass mymaster 123456" sentinel.conf

在其他服务器使用同样的sentinel.conf配置文件

2.3启动sentinel

redis-sentinel sentinel.conf 在三台服务器分别启动sentinel

[root@mycat redis-6.0.10]# redis-sentinel sentinel.conf
28085:X 17 Feb 2021 17:52:56.503 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
28085:X 17 Feb 2021 17:52:56.503 # Redis version=6.0.10, bits=64, commit=00000000, modified=0, pid=28085, just started
28085:X 17 Feb 2021 17:52:56.503 # Configuration loaded
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 6.0.10 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
 |    `-._   `._    /     _.-'    |     PID: 28085
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

28085:X 17 Feb 2021 17:52:56.505 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
28085:X 17 Feb 2021 17:52:56.511 # Sentinel ID is 2f928347f1644256ee2397a9a8c5bad3bb6d469c
28085:X 17 Feb 2021 17:52:56.511 # +monitor master mymaster 172.18.28.245 6379 quorum 2
28085:X 17 Feb 2021 17:52:56.512 * +slave slave 172.18.28.244:6379 172.18.28.244 6379 @ mymaster 172.18.28.245 6379
28085:X 17 Feb 2021 17:52:56.517 * +slave slave 172.18.28.243:6379 172.18.28.243 6379 @ mymaster 172.18.28.245 6379

2.4测试sentinel

登陆redis master服务器,查看主从集群信息

127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=172.18.28.244,port=6379,state=online,offset=43562,lag=0
slave1:ip=172.18.28.243,port=6379,state=online,offset=43562,lag=0
master_replid:52f3edd73c636afd95d42dbe7b770d7474fb9269
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:43562
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:43562
127.0.0.1:6379>

关闭master的redis服务

如图提示redis的245这台master服务关闭,然后新的 master的选择了243这台,查看集群信息:

[root@mycat ~]# redis-cli -h 172.18.28.243
172.18.28.243:6379> auth 123456
OK
172.18.28.243:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=172.18.28.244,port=6379,state=online,offset=144931,lag=0
master_replid:ec86e81d6691795b6f5cd21cc54379fc2aa552db
master_replid2:52f3edd73c636afd95d42dbe7b770d7474fb9269
master_repl_offset:144931
second_repl_offset:84014
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:144931
172.18.28.243:6379>

注意查看role:master表示现在登陆的243这台为master,而connected_slaves:1表示只有一台从服务器连接,因为挂掉的master服务器也变为从服务器了,但是没有启动

启动原245的master服务器的redis服务

提示连接到master 172.18.28.243:6379,说明主从已变。

[root@mycat redis-6.0.10]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:172.18.28.243
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0

这台服务器查看主从复制的集群信息,role:slave,表示是从服务器了。

master_host:172.18.28.243 说明了redis的主服务器 

在243的新master服务器查看主从集群信息:

172.18.28.243:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=172.18.28.244,port=6379,state=online,offset=237502,lag=1
slave1:ip=172.18.28.245,port=6379,state=online,offset=237643,lag=1
master_replid:ec86e81d6691795b6f5cd21cc54379fc2aa552db
master_replid2:52f3edd73c636afd95d42dbe7b770d7474fb9269
master_repl_offset:237784

 和没挂掉的主从一样的状态了,一主两从。

2.5 sentinel 事件说明

·       +reset-master :主服务器已被重置。

·       +slave :一个新的从服务器已经被 Sentinel 识别并关联。

·       +failover-state-reconf-slaves :故障转移状态切换到了 reconf-slaves 状态。

·       +failover-detected :另一个 Sentinel 开始了一次故障转移操作,或者一个从服务器转换成了主服务器。

·       +slave-reconf-sent :领头(leader)的 Sentinel 向实例发送了 [SLAVEOF](/commands/slaveof.html) 命令,为实例设置新的主服务器。

·       +slave-reconf-inprog :实例正在将自己设置为指定主服务器的从服务器,但相应的同步过程仍未完成。

·       +slave-reconf-done :从服务器已经成功完成对新主服务器的同步。

·       -dup-sentinel :对给定主服务器进行监视的一个或多个 Sentinel 已经因为重复出现而被移除 —— 当 Sentinel 实例重启的时候,就会出现这种情况。

·       +sentinel :一个监视给定主服务器的新 Sentinel 已经被识别并添加。

·       +sdown :给定的实例现在处于主观下线状态。

·       -sdown :给定的实例已经不再处于主观下线状态。

·       +odown :给定的实例现在处于客观下线状态。

·       -odown :给定的实例已经不再处于客观下线状态。

·       +new-epoch :当前的纪元(epoch)已经被更新。

·       +try-failover :一个新的故障迁移操作正在执行中,等待被大多数 Sentinel 选中(waiting to be elected by the majority)。

·       +elected-leader :赢得指定纪元的选举,可以进行故障迁移操作了。

·       +failover-state-select-slave :故障转移操作现在处于 select-slave 状态 —— Sentinel 正在寻找可以升级为主服务器的从服务器。

·       no-good-slave :Sentinel 操作未能找到适合进行升级的从服务器。Sentinel 会在一段时间之后再次尝试寻找合适的从服务器来进行升级,又或者直接放弃执行故障转移操作。

·       selected-slave :Sentinel 顺利找到适合进行升级的从服务器。

·       failover-state-send-slaveof-noone :Sentinel 正在将指定的从服务器升级为主服务器,等待升级功能完成。

·       failover-end-for-timeout :故障转移因为超时而中止,不过最终所有从服务器都会开始复制新的主服务器(slaves will eventually be configured to replicate with the new master anyway)。

·       failover-end :故障转移操作顺利完成。所有从服务器都开始复制新的主服务器了。

·       +switch-master :配置变更,主服务器的 IP 和地址已经改变。 这是绝大多数外部用户都关心的信息。

·       +tilt :进入 tilt 模式。

·       -tilt :退出 tilt 模式。

三、Cluster模式

主从集群--解决了一般的读写问题

sentinel集群+主从集群--解决了单个主从集群的master挂掉后的高可用问题

Cluster集群则上在上面的基础上增加了分片(分库)功能,解决单机redis的存储问题

3.1Cluster集群特点

多点redis节点互联,网络共享

所有节点都是一主一从(或一主多从),其中从不提供服务,仅作为备用

支持在线增加,删除节点

客户端可以连接任何一个主节点读写。

3.2Cluster搭建

cluster可以说是sentinel和主从模式的结合体,通过cluster可以实现主从和master重选功能,所以如果配置两个副本三个分片的话,就需要六个Redis实例

还是上面的三台机器,分别开启两个redis服务(端口),3台机器就是3个分片(分库),所以一共是3*2=6个redis服务。

172.18.28.245   端口7001,70002
172.18.28.244   端口7003,70004
172.18.28.243   端口7005,70006
同时防火墙端口要打开17001,17002,17003,17004,17005,17006
[root@mycat redis-6.0.10]# cp redis.conf.bak redis_7001.conf
[root@mycat redis-6.0.10]# cp redis.conf.bak redis_7002.conf
[root@mycat redis-6.0.10]# pwd
/usr/local/redis-6.0.10

复制两个原始的配置文件

 mkdir -p /data/redis/cluster/redis_7001
 mkdir -p /data/redis/cluster/redis_7002
 mkdir -p /data/redis/cluster/redis_7003
 mkdir -p /data/redis/cluster/redis_7004
 mkdir -p /data/redis/cluster/redis_7005
 mkdir -p /data/redis/cluster/redis_7006
 日志存放目录
 mkdir -p /usr/local/redis-6.0.10/cluster/

redis_7001.conf修改

 

bind 127.0.0.1  改成bind 127.0.0.1 172.18.28.245  可以绑定多个ip地址,这里选择localhost地址以及内网地址

port 6379 改成port 7001

daemonize no改成daemonize yes  #Redis will write a pid file in /var/run/redis.pid when daemonized,启动守护进程方式时会写一个pidfile文件

logfile "" 改为logfile "/usr/local/redis-6.0.10/cluster/redis_7001.log"

修改的脚本

#!/bin.bash
#$1==filename,$2==port
if [ -n $1 ];then
filename=$1
fi
filepath=/usr/local/redis-6.0.10
file=$filepath/$filename
Port=6379
if [ -f "$file" ];then
echo "exist"
else
echo "not exist"
exit
fi
#set the port,default 6379
if [ -n $2 ];then
Port=$2
fi
sed -i "s/port 6379/port $Port/g" $file

IP=`ip addr|awk '{if($2~"172")print $2}'|awk -F '/' '{print $1}'`
clusterDataPath="\/data\/redis\/cluster\/redis_$Port"
sed -i "s/^bind 127.0.0.1/bind $IP/g" $file

sed -i 's/protected-mode yes/protected-mode no/g' $file
sed -i "s/redis_6379/redis_$Port/g" $file
sed -i "s/logfile \"\"/logfile \/usr\/local\/redis-6.0.10\/cluster\/redis_${Port}.log/g" $file
sed -i "s/dir \.\//dir $clusterDataPath/" $file

sed -i 's/appendonly no/appendonly yes/g' $file
# requirepass foobared
sed -i "/# requirepass foobared/a requirepass 123456" $file
# masterauth <master-password>
sed -i "/# masterauth <master-password>/a masterauth 123456" $file
sed -i "/# cluster-enabled yes/a cluster-enabled yes" $file
sed -i "/# cluster-config-file nodes-6379.conf/a cluster-config-file nodes-${Port}.conf" $file
# cluster-config-file nodes-6379.conf
# cluster-node-timeout 15000
sed -i "/# cluster-node-timeout 15000/a cluster-node-timeout 15000" $file
sed -i "s/daemonize no/daemonize yes/g" $file

 执行sh clusterConfig.sh redis_7001.conf 7001 根据不同的端口号修改生成上面所需要的参数,只要维护一个脚本 文件,不用一个一个改。

sh clusterConfig.sh redis_7001.conf 7001

sh clusterConfig.sh redis_7002.conf 7002

sh clusterConfig.sh redis_7003.conf 7003

sh clusterConfig.sh redis_7004.conf 7004

sh clusterConfig.sh redis_7005.conf 7005

sh clusterConfig.sh redis_7006.conf 7006

[root@mycat redis-6.0.10]# sh clusterConfig.sh redis_7001.conf 7001
exist
[root@mycat redis-6.0.10]#

 

redis-cli -a 123456 --cluster create 172.18.28.245:7001 172.18.28.245:7002 172.18.28.244:7003 172.18.28.244:7004 172.18.28.243:7005 172.18.28.243:7006 --cluster-replicas 1  

[root@mycat redis-6.0.10]# redis-cli -a 123456 --cluster create 172.18.28.245:7001 172.18.28.245:7002 172.18.28.244:7003 172.18.28.244:7004 172.18.28.243:7005 172.18.28.243:7006 --cluster-replicas 1  
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.18.28.244:7004 to 172.18.28.245:7001
Adding replica 172.18.28.243:7006 to 172.18.28.244:7003
Adding replica 172.18.28.245:7002 to 172.18.28.243:7005
M: fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001
   slots:[0-5460] (5461 slots) master
S: 127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002
   replicates e25e8945bc8dfa06b5fd7670ceae790d93ab58ed
M: 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003
   slots:[5461-10922] (5462 slots) master
S: 2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004
   replicates fa92015aa467aab2c2966237093bec6c10943172
M: e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005
   slots:[10923-16383] (5461 slots) master
S: b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006
   replicates 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.18.28.245:7001)
M: fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004
   slots: (0 slots) slave
   replicates fa92015aa467aab2c2966237093bec6c10943172
S: 127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002
   slots: (0 slots) slave
   replicates e25e8945bc8dfa06b5fd7670ceae790d93ab58ed
S: b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006
   slots: (0 slots) slave
   replicates 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b
M: 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

显示集群搭建成功。

3.3 Cluster集群测试

登陆集群:redis-cli -c -h 172.18.28.245 -p 7001 -a 123456

查看集群信息:cluster info

列出节点信息:cluster nodes

[root@mycat redis-6.0.10]#  redis-cli -c -h 172.18.28.245 -p 7001 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.28.245:7001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:24441
cluster_stats_messages_pong_sent:24548
cluster_stats_messages_sent:48989
cluster_stats_messages_ping_received:24543
cluster_stats_messages_pong_received:24441
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:48989
172.18.28.245:7001>
172.18.28.245:7001> cluster nodes
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613677838498 5 connected 10923-16383
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 slave fa92015aa467aab2c2966237093bec6c10943172 0 1613677837000 1 connected
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613677840504 5 connected
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613677839000 3 connected
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 myself,master - 0 1613677839000 1 connected 0-5460
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613677839501 3 connected 5461-10922
172.18.28.245:7001>
 

写入数据验证分片功能

172.18.28.245:7001> set foo bar1
-> Redirected to slot [12182] located at 172.18.28.243:7005   说明数据到了7005,172.18.28.243:7005
OK
172.18.28.243:7005> set foo bar2
OK
172.18.28.243:7005> set foo bar3
OK
172.18.28.243:7005> set foo1 bar1
OK
172.18.28.243:7005> set foo2 bar2
-> Redirected to slot [1044] located at 172.18.28.245:7001   同上
OK
172.18.28.245:7001> set foo3 bar3
OK
172.18.28.245:7001> set foo4 bar4
-> Redirected to slot [9426] located at 172.18.28.244:7003    同上
OK
172.18.28.244:7003>
172.18.28.244:7003> get foo   在7003上执行get foo,foo是写在172.18.28.243:7005上的,所以自动转到172.18.28.243:7005获取这个值。
-> Redirected to slot [12182] located at 172.18.28.243:7005
"bar3"
172.18.28.243:7005>

可以看出redis cluster集群是去中心化的,每个节点都是平等的,连接哪个节点都可以获取和设置数据。

增加节点,增加的节点角色默认都是master

redis_7007.conf在245服务器,redis_7008.conf在244服务器

[root@mycat redis-6.0.10]# redis-cli -c -h 172.18.28.245 -p 7001
172.18.28.245:7001> auth 123456
OK
172.18.28.245:7001> cluster meet 172.18.28.245 7007
OK

172.18.28.245:7001> cluster meet 172.18.28.244 7008
OK
172.18.28.245:7001> cluster nodes
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 master - 0 1613679599000 7 connected
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613679599000 5 connected 10923-16383
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 slave fa92015aa467aab2c2966237093bec6c10943172 0 1613679599000 1 connected
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613679601793 5 connected
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613679599000 3 connected
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 myself,master - 0 1613679595000 1 connected 0-5460
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 master - 0 1613679600000 0 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613679600791 3 connected 5461-10922
172.18.28.245:7001>

在集群中将7008改为7007的slave

172.18.28.245:7001> cluster nodes
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 master - 0 1613679981663 7 connected
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613679981000 5 connected 10923-16383
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 slave fa92015aa467aab2c2966237093bec6c10943172 0 1613679979000 1 connected
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613679982665 5 connected
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613679981000 3 connected
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 myself,master - 0 1613679983000 1 connected 0-5460
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 master - 0 1613679983668 0 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613679984671 3 connected 5461-10922
172.18.28.245:7001> cluster replicate
[root@mycat redis-6.0.10]# redis-cli -c -h 172.18.28.244 -p 7008 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.18.28.244:7008> cluster replicate 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7
OK
172.18.28.244:7008> cluster nodes
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 master - 0 1613680071028 7 connected
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613680069000 5 connected
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613680074037 5 connected 10923-16383
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 slave fa92015aa467aab2c2966237093bec6c10943172 0 1613680071000 1 connected
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 master - 0 1613680069000 1 connected 0-5460
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 myself,slave 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 0 1613680070000 7 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613680073034 3 connected 5461-10922
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613680072031 3 connected
172.18.28.244:7008>
如红色部分所示,7008的redis服务已成功7007redis服务的slave服务。

删除节点

172.18.28.244:7008> cluster nodes
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 master - 0 1613680219000 7 connected
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613680216000 5 connected
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613680217335 5 connected 10923-16383
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 slave fa92015aa467aab2c2966237093bec6c10943172 0 1613680221000 1 connected
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 master - 0 1613680220341 1 connected 0-5460
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 myself,slave 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 0 1613680219000 7 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613680218000 3 connected 5461-10922
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613680221343 3 connected
172.18.28.244:7008> cluster forget a57ee406e9cb504babacdcd63ed8b2194029e316
(error) ERR I tried hard but I can't forget myself...  不能删除自己的登陆节点
172.18.28.244:7008> cluster forget 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7
(error) ERR Can't forget my master!  也不能删除登陆节点对应的master节点
 

成功删除节点:

172.18.28.244:7008> cluster forget fa92015aa467aab2c2966237093bec6c10943172
OK
172.18.28.244:7008> cluster forget 127e48427a23769f23c09df11b65d56db1e25938
OK
172.18.28.244:7008> cluster nodes
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 master - 0 1613680492892 7 connected
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613680493000 5 connected 10923-16383
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 slave - 0 1613680494896 1 connected
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 myself,slave 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 0 1613680492000 7 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613680491000 3 connected 5461-10922
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613680494000 3 connected
172.18.28.244:7008>
过一会删除的节点自动恢复了

172.18.28.244:7008> cluster nodes
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613680558032 5 connected
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 master - 0 1613680556000 7 connected
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613680556000 5 connected 10923-16383
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 master - 0 1613680557000 1 connected 0-5460
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 slave fa92015aa467aab2c2966237093bec6c10943172 0 1613680553019 1 connected
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 myself,slave 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 0 1613680555000 7 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613680554021 3 connected 5461-10922
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613680557031 3 connected
172.18.28.244:7008>

ps -e
模拟7001的master挂掉,查看它的slave 7004集群服务状态

172.18.28.244:7008> cluster nodes
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613707480000 5 connected
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 master - 0 1613707481000 7 connected
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613707481464 5 connected 10923-16383
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 slave 2e500a178dffea762e15b2f92d5819041b5a2574 0 1613707482466 8 connected
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 master - 0 1613707483469 8 connected 0-5460
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 myself,slave 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 0 1613707482000 7 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613707480462 3 connected 5461-10922
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613707480000 3 connected
172.18.28.244:7008>

 如上代码所示,7004现在是master,而7001的redis重启之后变成了slave.

master挂掉会由slave自动变成master,原master重启后变成slave.

但是如果使用7007和7008来测试,7007master挂掉,7008并不会自动切换,

这是因为7007节点上根本没数据。集群数据被分为三份,采用哈希槽 (hash slot)的方式来分配16384个slot的话,它们三个节点分别承担的slot 区间是

节点7004覆盖0-5460
节点7003覆盖5461-10922
节点7005覆盖10923-16383

在集群创建时有提示。

那么增加的节点还有什么用?

3.4Cluster 伸缩

3.4.1 扩容--slot分配

对主节点重新分配slot

新增加的主节点,是没有slots的,主节点如果没有slots的话,存取数据就都不会被选中
可以把分配的过程理解成打扑克牌,all表示大家重新洗牌;输入某个主节点的node id,然后在输入done的话,就好比从某个节点,抽牌:

redis-cli --cluster help

[root@mycat redis-6.0.10]# redis-cli --cluster reshard 172.18.28.245 7007 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 172.18.28.245:7007)
M: 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007
   slots: (0 slots) master
   1 additional replica(s)
S: fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001
   slots: (0 slots) slave
   replicates 2e500a178dffea762e15b2f92d5819041b5a2574
S: b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006
   slots: (0 slots) slave
   replicates 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b
M: 2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002
   slots: (0 slots) slave
   replicates e25e8945bc8dfa06b5fd7670ceae790d93ab58ed
M: 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008
   slots: (0 slots) slave
   replicates 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?

 给7007分配slot,总共16384个slot,加上7007一共4台master,平均slot=16384/4=4096

所以输入4096,然后要求输入node ID,直接输入all

分配完成后,再次执行分配 会出错。

可以看到7007,分别从原有的节点中分配了一小段slot:

[0-1364]后面接7004的slot

[5461-6826]后面接7003的slot

[10923-12287]后面接7005的slot

172.18.28.245:7007> cluster nodes
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 slave 2e500a178dffea762e15b2f92d5819041b5a2574 0 1613710623517 8 connected
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613710622000 3 connected
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 master - 0 1613710623000 8 connected 1365-5460
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613710625521 5 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613710623000 3 connected 6827-10922
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613710624518 5 connected 12288-16383
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 slave 28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 0 1613710625000 9 connected
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 myself,master - 0 1613710621000 9 connected 0-1364 5461-6826 10923-12287
172.18.28.245:7007>
至此,redis  cluster 扩容成功。

再次测试7007挂掉的高可用是否正常切换

成功切换并且slot也正确,启动7007,要将7007,7008的slot划到7003

redis-cli --cluster reshard 172.18.28.244 7008 -a 123456

3.4.2 缩容--slot分配

计划将7007和7008下线

确认分配slot计划

执行分配slot计划完成,查看节点信息,里面有slot分布。然后可以将7007和7008下线了。

172.18.28.245:7001> cluster nodes
fa92015aa467aab2c2966237093bec6c10943172 172.18.28.245:7001@17001 myself,slave 2e500a178dffea762e15b2f92d5819041b5a2574 0 1613712605000 8 connected
a57ee406e9cb504babacdcd63ed8b2194029e316 172.18.28.244:7008@17008 master - 0 1613712608000 10 connected
5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 172.18.28.244:7003@17003 master - 0 1613712609489 11 connected 0-1364 5461-12287
b75ca6573efa61300316f501c2fda0e3dbd7aee0 172.18.28.243:7006@17006 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613712610000 11 connected
e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 172.18.28.243:7005@17005 master - 0 1613712610492 5 connected 12288-16383
2e500a178dffea762e15b2f92d5819041b5a2574 172.18.28.244:7004@17004 master - 0 1613712611000 8 connected 1365-5460
127e48427a23769f23c09df11b65d56db1e25938 172.18.28.245:7002@17002 slave e25e8945bc8dfa06b5fd7670ceae790d93ab58ed 0 1613712611495 5 connected
28e8dadc06b30b7a4dc3b3b536b388ba3f0389b7 172.18.28.245:7007@17007 slave 5e1834ac9b5c3798af4c850403ec0c3d8363ac9b 0 1613712608489 11 connected
172.18.28.245:7001>

未完待续。。。

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值