NoSQL-redis

redis介绍

特点:
1、速度快
2、多种数据结构
3、功能丰富
4、简单稳定
5、支持客户端语言多
6、支持数据持久化
7、自带高可用架构

Redis使用场景:
1、键过期功能:缓存,session会话保存,优惠卷过期
2、列表:排行榜
3、天然计数器:帖子浏览数,视频播放数,评论留言数
4、集合:兴趣标签,广告投放
5、消息队列:ELK

redis安装部署

  1. 目录规划

#redis下载目录
/data/soft/
#redis安装目录
/opt/redis_cluster/redis_{PORT}/{conf,logs.pid}
#redis数据目录
/opt/redis_cluster/redis_{PORT}/redis_{PORT}.rdb

  1. 创建目录
[root@localhost ~]# mkdir /data/soft/
[root@localhost ~]# mkdir -p /opt/redis_cluster/redis_6379/{conf,logs,pid}
  1. 下载安装redis
[root@localhost redis_6379]# cd /data/soft/
[root@localhost soft]# wget http://download.redis.io/releases/redis-3.2.12.tar.gz
[root@localhost soft]# tar -xzvf redis-3.2.12.tar.gz
[root@localhost soft]# mv redis-3.2.12 /opt/redis_cluster/
[root@localhost soft]# cd /opt/redis_cluster/
[root@localhost redis_cluster]# ln -sf redis-3.2.12 redis
[root@localhost redis]# yum -y install gcc automake autoconf libtool make
[root@localhost redis]# make && make install
  1. 配置redis
[root@localhost redis]# /opt/redis_cluster/redis_6379/conf
[root@localhost conf]# vim 6379.conf

# 以守护进程模式启动
daemonize yes
#绑定的主机地址
bind 192.168.168.132
# 监听端口
port 6379
# pid文件和log文件的保存地址
pidfile /opt/redis_cluster/redis_6379/pid/redis_6379.pid
logfile /opt/redis_cluster/redis_6379/logs/redis_6379.log
#设置数据库的数量,默认数据库为0
databases 16
# 本地数据库的目录
dir /opt/redis_cluster/redis_6379/
# 指定本地持久化文件的文件名,默认是dump.rdb
dbfilename redis_6379.rdb
  1. 启动redis
[root@localhost conf]# redis-server /opt/redis_cluster/redis_6379/conf/6379.conf 
[root@localhost conf]# ps aux|grep redis
root      12484  0.0  0.1 136972  2064 ?        Ssl  22:10   0:00 redis-server 192.168.168.132:6379
  1. 停redis
[root@localhost conf]# redis-cli -h 192.168.168.132
192.168.168.132:6379> SHUTDOWN

redis基本操作命令

Redis数据类型

数据类型介绍
String字符类型
Hash字典类型
List列表
Set集合
Sorted set有序集合

全局操作

  1. 查看所有命键
Keys *
**十分危险的命令,线上禁止使用**
  1. 返回键所存储值的类型
TYPE key
  1. 判断该键是否存在,存在返回 1,否则返回0
192.168.168.132:6379> exists name
(integer) 1
192.168.168.132:6379> exists work
(integer) 0
  1. 删除已有得键
192.168.168.132:6379> DEL age
(integer) 1
192.168.168.132:6379> exists age
(integer) 0

Strings操作

  1. 设置一个key并赋值
set key value
  1. 查看一个key
get key
  1. 计数器
    每点一次关注,都执行以下命令一次
incr num  #每次执行一下都会加1,后面如果跟key,key必须得是数字
get num 
  1. 增加数值
    暗箱操作,例如直接增加,关注人数
192.168.168.132:6379> get num
"15"
192.168.168.132:6379> INCRBY num 10000
(integer) 10015
192.168.168.132:6379> get num
"10015"
192.168.168.132:6379> DECRBY num 10000
(integer) 15
192.168.168.132:6379> get num
"15"
  1. 设置多个key并赋值
192.168.168.132:6379> MSET id 101 name xxx age 20
OK
192.168.168.132:6379> get id
"101"
192.168.168.132:6379> get name
"xxx"
192.168.168.132:6379> get age
"20"
  1. 查看多个key
192.168.168.132:6379> MGET id name age
1) "101"
2) "xxx"
3) "20"
  1. 设置过期时间
192.168.168.132:6379> TTL num
(integer) -1
192.168.168.132:6379> EXPIRE num 10
(integer) 1
192.168.168.132:6379> TTL num
(integer) 9
192.168.168.132:6379> TTL num
(integer) 1
192.168.168.132:6379> TTL num
(integer) -2
192.168.168.132:6379> EXISTS num
(integer) 0

TTL查看过期时间的三种值
-1:永不过期
-2:没有这个key
数字:还有多少秒过期
注意:假如一个key过期时间是100秒,如果在过期时间期间,给key重新赋值,TTL查看这个key就会变成-1

实验案例

192.168.168.132:6379> EXPIRE age 100
(integer) 1
192.168.168.132:6379> TTL age
(integer) 93
192.168.168.132:6379> TTL age
(integer) 92
192.168.168.132:6379> TTL age
(integer) 87
192.168.168.132:6379> set age 18
OK
192.168.168.132:6379> TTL age
(integer) -1
  1. 取消过期时间
192.168.168.132:6379> EXPIRE age 100
(integer) 1
192.168.168.132:6379> TTL age
(integer) 97
192.168.168.132:6379> TTL age
(integer) 96
192.168.168.132:6379> PERSIST age
(integer) 1
192.168.168.132:6379> TTL age
(integer) -1

List列表的操作

  1. 从右边开始插入列表,也就是从列表最后开始插入
192.168.168.132:6379> RPUSH list1 a b c
(integer) 3
192.168.168.132:6379> TYPE list1
list
  1. 从左边开始插入列表,也就是从列表开头开始插入
192.168.168.132:6379> LPUSH list1 1 2 3
(integer) 3
  1. 查看列表有多长
192.168.168.132:6379> LLEN list1
(integer) 6
  1. 查看列表内容
192.168.168.132:6379> LRANGE list1 0 -1
1) "3"
2) "2"
3) "1"
4) "a"
5) "b"
6) "c"

从0开始,-1代表最后一个,-2就是最后第二个,以此内推

  1. 从右边开始删除第一个值,及删除最后一个值
192.168.168.132:6379> RPOP list1
"c"
192.168.168.132:6379> LRANGE list1 0 -1
1) "3"
2) "2"
3) "1"
4) "a"
5) "b"
  1. 从左边开始删除第一个值,及删除最后一个值
192.168.168.132:6379> LPOP list1
"3"
192.168.168.132:6379> LRANGE list1 0 -1
1) "2"
2) "1"
3) "a"
4) "b"

Hash字典类型操作

192.168.168.132:6379> HMSET user:100 name xx age 20 job it
OK
192.168.168.132:6379> HMGET user:100 name
1) "xx"
192.168.168.132:6379> HMGET user:100 name age job
1) "xx"
2) "20"
3) "it"
192.168.168.132:6379> HGETALL user:100
1) "name"
2) "xx"
3) "age"
4) "20"
5) "job"
6) "it"

集合类型操作

set集合不允许出现重复的元素

#添加集合
192.168.168.132:6379> SADD set1 1 2 3 4 5
(integer) 5
192.168.168.132:6379> SADD set2 3 4 5 6 7
(integer) 5
#查看集合内容
192.168.168.132:6379> SMEMBERS set1
1) "1"
2) "2"
3) "3"
4) "4"
5) "5"
192.168.168.132:6379> SMEMBERS set2
1) "3"
2) "4"
3) "5"
4) "6"
5) "7"
#集合的不同的
192.168.168.132:6379> SDIFF set1 set2
1) "1"
2) "2"
#集合的相同的
192.168.168.132:6379> SINTER set1 set2
1) "3"
2) "4"
3) "5"
#集合的并集
192.168.168.132:6379> SUNION set1 set2
1) "1"
2) "2"
3) "3"
4) "4"
5) "5"
6) "6"
7) "7"

redis持久化

如果不做redis持久化,在重启redis的时候,插入redis的数据将都会被清空

RDB持久化

可以在指定的时间间隔内生成数据集的 时间点快照(point-in-time snapshot)。
优点:速度快,适合于用做备份,主从复制也是基于RDB持久化功能实现的。
缺点:会有数据丢失

  • 手动生成备份文件
[root@localhost ~]# redis-cli -h 192.168.168.132
192.168.168.132:6379> BGSAVE
Background saving started
192.168.168.132:6379> quit
[root@localhost ~]# cd /opt/redis_cluster/redis_6379/
[root@localhost redis_6379]# ls
conf  logs  pid  redis_6379.rdb
#redis_6379.rdb就是生成的数据文件
  • 在配置文件中配置
rdb持久化核心配置参数:
vim /opt/redis_cluster/redis_6379/conf/6379.conf
dir /opt/redis_cluster/redis_6379/
dbfilename redis_6379.rdb
save 900 1
save 300 10
save 60 10000

#配置分别表示:
#900秒(15分钟)内有1个更改
#300秒(5分钟)内有10个更改
#60秒内有10000个更改

当执行SHUTDOWN,相当于先执行BGSAVE再SHUTDOWN,会先将redis_6379.rdb文件更新再停掉程序。如果非正常关闭的话,redis_6379.rdb文件不会有变化如果不满足于配置文件中save的配置,那数据将会丢失

AOF持久化

记录服务器执行的所有写操作命令,并在服务器启动时,通过重新执行这些命令来还原数据集。
AOF 文件中的命令全部以 Redis 协议的格式来保存,新命令会被追加到文件的末尾。
优点:可以最大程度保证数据不丢
缺点:日志记录量级比较大

配置AOF

[root@localhost ~]# vim /opt/redis_cluster/redis_6379/conf/6379.conf
#是否打开aof日志功能
appendonly yes     
#每1个命令,都立即同步到aof 
appendfsync always  
#每秒写1次
appendfsync everysec 
#写入工作交给操作系统,由操作系统判断缓冲区大小,统一写入到aof.
appendfsync no
appendfilename "append6379.aof"

注意:如果AOF和RDB的备份文件同时存在,会优先读取AOF,即使AOF数据不完整也会优先读取AOF

redis安全认证

redis默认开启了保护模式,只允许配置文件中的Bind配置的地址登录并访问数据库。
禁止protected-mode
protected-mode yes/no (保护模式,是否开启)

  1. 配置安装认证
[root@localhost redis_6379]# vim /opt/redis_cluster/redis_6379/conf/6379.conf
#增加密码配置
requirepass 123
  1. 验证
[root@localhost redis_6379]# redis-cli -h 192.168.168.132 -a 123
192.168.168.132:6379> get k1
"v1"
192.168.168.132:6379> quit
[root@localhost redis_6379]# redis-cli -h 192.168.168.132
192.168.168.132:6379> AUTH 123
OK
192.168.168.132:6379> get k1
"v1"
192.168.168.132:6379> quit

redis主从复制

主从复制流程
1、从库发起同步请求
2、主库收到请求后执行BGSAVE,将内存里的数据保存到磁盘
3、主库将持久化的数据发送给从库的数据目录
4、从库收到主库的持久化数据后,先清空当前自己内存中的所有数据
5、从库加载数据目录的持久化文件

  1. 环境准备

准备两个或两个以上redis实例
主节点:6380
从节点:6381、6382

[root@localhost redis_cluster]# mkdir /data/redis_cluster/redis_6380
[root@localhost redis_cluster]# mkdir /data/redis_cluster/redis_6381
[root@localhost redis_cluster]# mkdir /data/redis_cluster/redis_6382
[root@localhost ~]# cd /opt/redis_cluster/
[root@localhost redis_cluster]# cp -r redis_6379 redis_6380
[root@localhost redis_cluster]# cp -r redis_6379 redis_6381
[root@localhost redis_cluster]# cp -r redis_6379 redis_6382
[root@localhost redis_cluster]# mv redis_6380/conf/6379.conf redis_6380/conf/6380.conf
[root@localhost redis_cluster]# sed -i 's/6379/6381/' redis_6381/conf/6379.conf
[root@localhost redis_cluster]# mv redis_6381/conf/6379.conf redis_6381/conf/6381.conf
[root@localhost redis_cluster]# sed -i 's/6379/6382/' redis_6382/conf/6379.conf
[root@localhost redis_cluster]# mv redis_6382/conf/6379.conf redis_6382/conf/6382.conf
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6380/conf/6380.conf 
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6381/conf/6381.conf 
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6382/conf/6382.conf
[root@localhost redis_cluster]# ps aux|grep redis
root      38267  0.0  0.1 136972  3064 ?        Ssl  10:44   0:00 redis-server 192.168.168.132:6379
root      38384  0.0  0.1 136972  2064 ?        Ssl  10:51   0:00 redis-server 192.168.168.132:6380
root      38388  0.0  0.1 136972  2064 ?        Ssl  10:51   0:00 redis-server 192.168.168.132:6381
root      38392  0.0  0.1 136972  2068 ?        Ssl  10:52   0:00 redis-server 192.168.168.132:6382
  1. 配置主从同步
[root@localhost redis_6380]# redis-cli  -h 192.168.168.132 -p 6381
192.168.168.132:6381> SLAVEOF 192.168.168.132 6380
OK
192.168.168.132:6381> quit
[root@localhost redis_6380]# redis-cli  -h 192.168.168.132 -p 6382
192.168.168.132:6382> SLAVEOF 192.168.168.132 6380
OK
192.168.168.132:6382> quit
  1. 查看主从复制的状态
[root@localhost redis_6380]# redis-cli  -h 192.168.168.132 -p 6380 info replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.168.132,port=6381,state=online,offset=113,lag=0
slave1:ip=192.168.168.132,port=6382,state=online,offset=113,lag=0
master_repl_offset:113
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:112
[root@localhost redis_6380]# redis-cli  -h 192.168.168.132 -p 6381 info replication
# Replication
role:slave
master_host:192.168.168.132
master_port:6380
master_link_status:up
master_last_io_seconds_ago:9
master_sync_in_progress:0
slave_repl_offset:127
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
[root@localhost redis_6380]# redis-cli  -h 192.168.168.132 -p 6382 info replication
# Replication
role:slave
master_host:192.168.168.132
master_port:6380
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:141
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
  1. 写入配置文件,使重启redis也可以自己建立主从关系
[root@localhost ~]# echo 'SLAVEOF 192.168.168.132 6380' >> /opt/redis_cluster/redis_6381/conf/6381.conf 
[root@localhost ~]# echo 'SLAVEOF 192.168.168.132 6380' >> /opt/redis_cluster/redis_6382/conf/6382.conf

redis哨兵

哨兵的主要功能
1、监控
2、提醒
3、自动处理故障节点
哨兵需要在每台主机上安装

  1. 创建目录
mkdir -p /data/redis_cluster/redis_26380/
mkdir -p /opt/redis_cluster/redis_26380
cd /opt/redis_cluster/redis_26380
mkdir conf logs pid
cd conf
  1. 编写配置文件
[root@localhost conf]# vim 26380.conf 

# 以守护进程模式启动
daemonize yes
#绑定的主机地址
bind 192.168.168.132
# 监听端口
port 26380
# pid文件和log文件的保存地址
pidfile /opt/redis_cluster/redis_26380/pid/redis_26380.pid
logfile /opt/redis_cluster/redis_26380/logs/redis_26380.log
# 本地数据库的目录
dir /data/redis_cluster/redis_26380/
#主节点的IP和端口,最后那个1是当主节点挂了,几个sentinel参与选举,我这里是一台主机所以写1,一般都是三台主机的话,就是两个
sentinel monitor mymaster 192.168.168.132 6380 1
#指定sentinel认为服务器断线的毫秒数
sentinel down-after-milliseconds mymaster 5000
#向新主节点发起复制操作,配置为1是,1个1个轮询操作
sentinel parallel-syncs mymaster 1
#故障转移超时时间
sentinel failover-timeout mymaster 18000
  1. 启动哨兵节点
redis-sentinel /opt/redis_cluster/redis_26380/conf/26380.conf
[root@localhost conf]# ps aux|grep redis
root      38267  0.0  0.1 136972  3064 ?        Ssl  10:44   0:13 redis-server 192.168.168.132:6379
root      38384  0.0  0.1 139020  3396 ?        Ssl  10:51   0:14 redis-server 192.168.168.132:6380
root      38388  0.0  0.1 139020  3512 ?        Ssl  10:51   0:13 redis-server 192.168.168.132:6381
root      38392  0.0  0.1 139020  3304 ?        Ssl  10:52   0:13 redis-server 192.168.168.132:6382
root      50999  0.1  0.1 136972  2268 ?        Ssl  17:17   0:00 redis-sentinel 192.168.168.132:26380 [sentinel]
  1. 验证
[root@localhost conf]#  redis-cli  -h 192.168.168.132 -p 6380
192.168.168.132:6380> SHUTDOWN
not connected> quit
[root@localhost conf]# redis-cli -h 192.168.168.132 -p 6381  info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.168.132,port=6382,state=online,offset=5197,lag=1
master_repl_offset:5197
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:5196
[root@localhost conf]# redis-server /opt/redis_cluster/redis_6380/conf/6380.conf
[root@localhost conf]# redis-cli -h 192.168.168.132 -p 6381  info replication
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.168.132,port=6382,state=online,offset=14102,lag=0
slave1:ip=192.168.168.132,port=6380,state=online,offset=14088,lag=1
master_repl_offset:14247
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:14246
  1. 通过哨兵模式强制选取主节点
    通过修改权重,然后进入哨兵强制换主节点
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6381
192.168.168.132:6381> CONFIG SET slave-priority 0
OK
192.168.168.132:6381> quit
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6382
192.168.168.132:6382> CONFIG SET slave-priority 0
OK
192.168.168.132:6382>
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 26380
192.168.168.132:26380> sentinel failover mymaster
OK
192.168.168.132:26380> SENTINEL get-master-addr-by-name mymaster
1) "192.168.168.132"
2) "6380"
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6381 CONFIG SET slave-priority 100
OK
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6382 CONFIG SET slave-priority 100
OK
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6381 CONFIG GET slave-priority
1) "slave-priority"
2) "100"
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6382 CONFIG GET slave-priority
1) "slave-priority"
2) "100"

Redis Cluster

一个集群有16384个槽位 0-16383
数据分片入,假如有三个redis集群,数据会按照16384个槽位平均分到三个集群,但不是按序号顺序分的
只要有一个槽位有问题或者没分配,整个集群不可使用

优秀的拓扑图

我这里实验用的多实例,一般工作中三台服务器,交叉做主从,这样当一台服务器挂了,不会影响数据
在这里插入图片描述

搭建集群

  1. 创建6个redis实例
[root@localhost conf]# mkdir /data/redis_cluster/redis_6390
[root@localhost conf]# mkdir /data/redis_cluster/redis_6391
[root@localhost conf]# mkdir /data/redis_cluster/redis_6392
[root@localhost conf]# mkdir /data/redis_cluster/redis_6393
[root@localhost conf]# mkdir /data/redis_cluster/redis_6394
[root@localhost conf]# mkdir /data/redis_cluster/redis_6395
[root@localhost redis_cluster]# pwd
/opt/redis_cluster
[root@localhost redis_cluster]# mkdir -p redis_6390/{conf,logs,pid}
[root@localhost conf]# vim redis_cluster.conf

bind 192.168.168.132
port 6390
daemonize yes
pidfile "/opt/redis_cluster/redis_6390/pid/redis_6390.pid"
logfile "/opt/redis_cluster/redis_6390/logs/redis_6390.log"
dbfilename "redis_6390.rdb"
dir "/data/redis_cluster/redis_6390/"
#激活集群模式
cluster-enabled yes
#集群的配置文件
cluster-config-file nodes_6390.conf
#集群的超时时间
cluster-node-timeout 15000
[root@localhost redis_cluster]# cp -r redis_6390 redis_6391
[root@localhost redis_cluster]# cp -r redis_6390 redis_6392
[root@localhost redis_cluster]# cp -r redis_6390 redis_6393
[root@localhost redis_cluster]# cp -r redis_6390 redis_6394
[root@localhost redis_cluster]# cp -r redis_6390 redis_6395
[root@localhost redis_cluster]# sed -i 's/6390/6391/g' redis_6391/conf/redis_cluster.conf 
[root@localhost redis_cluster]# sed -i 's/6390/6392/g' redis_6392/conf/redis_cluster.conf 
[root@localhost redis_cluster]# sed -i 's/6390/6393/g' redis_6393/conf/redis_cluster.conf 
[root@localhost redis_cluster]# sed -i 's/6390/6394/g' redis_6394/conf/redis_cluster.conf 
[root@localhost redis_cluster]# sed -i 's/6390/6395/g' redis_6395/conf/redis_cluster.conf
  1. 启动
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6390/conf/redis_cluster.conf
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6391/conf/redis_cluster.conf
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6392/conf/redis_cluster.conf
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6393/conf/redis_cluster.conf
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6394/conf/redis_cluster.conf
[root@localhost redis_cluster]# redis-server /opt/redis_cluster/redis_6395/conf/redis_cluster.conf
[root@localhost redis_cluster]# ps aux|grep redis
root      54155  0.0  0.1 136972  2508 ?        Ssl  15:54   0:00 redis-server 192.168.168.132:6390 [cluster]
root      54159  0.0  0.1 136972  2504 ?        Ssl  15:54   0:00 redis-server 192.168.168.132:6391 [cluster]
root      54163  0.0  0.1 136972  2504 ?        Ssl  15:54   0:00 redis-server 192.168.168.132:6392 [cluster]
root      54167  0.1  0.1 136972  2500 ?        Ssl  15:54   0:00 redis-server 192.168.168.132:6393 [cluster]
root      54171  0.0  0.1 136972  2508 ?        Ssl  15:54   0:00 redis-server 192.168.168.132:6394 [cluster]
root      54176  0.0  0.1 136972  2504 ?        Ssl  15:54   0:00 redis-server 192.168.168.132:6395 [cluster]

配置节点发现

查看集群节点

[root@localhost redis_6390]# cat /data/redis_cluster/redis_6390/nodes_6390.conf 
c335d8defa53ab7595d1b191feb575341f925240 :0 myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6390
192.168.168.132:6390> CLUSTER NODES
c335d8defa53ab7595d1b191feb575341f925240 :6390 myself,master - 0 0 0 connected
192.168.168.132:6390>

发现节点

192.168.168.132:6390> CLUSTER MEET 192.168.168.132 6391
OK
192.168.168.132:6390> CLUSTER MEET 192.168.168.132 6392
OK
192.168.168.132:6390> CLUSTER MEET 192.168.168.132 6393
OK
192.168.168.132:6390> CLUSTER MEET 192.168.168.132 6394
OK
192.168.168.132:6390> CLUSTER MEET 192.168.168.132 6395
OK
192.168.168.132:6390> CLUSTER NODES
c335d8defa53ab7595d1b191feb575341f925240 192.168.168.132:6390 myself,master - 0 0 1 connected
fa735c03c464753311abeb12b148390c715b09ad 192.168.168.132:6392 master - 0 1640334407474 0 connected
ad3a32dea36079f2620a3c83b2cad46ad34f458a 192.168.168.132:6394 master - 0 1640334405458 3 connected
9a935f9f3ab69813c7521e3d0300534b8e702138 192.168.168.132:6395 master - 0 1640334408483 5 connected
bee33ae4cadb6d222bb0cef4c58474aa8323124a 192.168.168.132:6393 master - 0 1640334409491 4 connected
f7922da8be6327bf979edc539c309e30d992d2a9 192.168.168.132:6391 master - 0 1640334406466 2 connected
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6391 CLUSTER NODES
f7922da8be6327bf979edc539c309e30d992d2a9 192.168.168.132:6391 myself,master - 0 0 2 connected
ad3a32dea36079f2620a3c83b2cad46ad34f458a 192.168.168.132:6394 master - 0 1640334448054 3 connected
9a935f9f3ab69813c7521e3d0300534b8e702138 192.168.168.132:6395 master - 0 1640334448559 5 connected
fa735c03c464753311abeb12b148390c715b09ad 192.168.168.132:6392 master - 0 1640334445026 0 connected
c335d8defa53ab7595d1b191feb575341f925240 192.168.168.132:6390 master - 0 1640334441995 1 connected
bee33ae4cadb6d222bb0cef4c58474aa8323124a 192.168.168.132:6393 master - 0 1640334443008 4 connected
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6392 CLUSTER NODES
9a935f9f3ab69813c7521e3d0300534b8e702138 192.168.168.132:6395 master - 0 1640334450966 5 connected
fa735c03c464753311abeb12b148390c715b09ad 192.168.168.132:6392 myself,master - 0 0 0 connected
f7922da8be6327bf979edc539c309e30d992d2a9 192.168.168.132:6391 master - 0 1640334446907 2 connected
c335d8defa53ab7595d1b191feb575341f925240 192.168.168.132:6390 master - 0 1640334449959 1 connected
bee33ae4cadb6d222bb0cef4c58474aa8323124a 192.168.168.132:6393 master - 0 1640334451975 4 connected
ad3a32dea36079f2620a3c83b2cad46ad34f458a 192.168.168.132:6394 master - 0 1640334448951 3 connected
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6393 CLUSTER NODES
fa735c03c464753311abeb12b148390c715b09ad 192.168.168.132:6392 master - 0 1640334454999 0 connected
ad3a32dea36079f2620a3c83b2cad46ad34f458a 192.168.168.132:6394 master - 0 1640334450966 3 connected
f7922da8be6327bf979edc539c309e30d992d2a9 192.168.168.132:6391 master - 0 1640334453990 2 connected
9a935f9f3ab69813c7521e3d0300534b8e702138 192.168.168.132:6395 master - 0 1640334452982 5 connected
bee33ae4cadb6d222bb0cef4c58474aa8323124a 192.168.168.132:6393 myself,master - 0 0 4 connected
c335d8defa53ab7595d1b191feb575341f925240 192.168.168.132:6390 master - 0 1640334451974 1 connected
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6394 CLUSTER NODES
9a935f9f3ab69813c7521e3d0300534b8e702138 192.168.168.132:6395 master - 0 1640334454898 5 connected
bee33ae4cadb6d222bb0cef4c58474aa8323124a 192.168.168.132:6393 master - 0 1640334457948 4 connected
f7922da8be6327bf979edc539c309e30d992d2a9 192.168.168.132:6391 master - 0 1640334455908 2 connected
fa735c03c464753311abeb12b148390c715b09ad 192.168.168.132:6392 master - 0 1640334456941 0 connected
ad3a32dea36079f2620a3c83b2cad46ad34f458a 192.168.168.132:6394 myself,master - 0 0 3 connected
c335d8defa53ab7595d1b191feb575341f925240 192.168.168.132:6390 master - 0 1640334454392 1 connected
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6395 CLUSTER NODES
bee33ae4cadb6d222bb0cef4c58474aa8323124a 192.168.168.132:6393 master - 0 1640334457646 4 connected
9a935f9f3ab69813c7521e3d0300534b8e702138 192.168.168.132:6395 myself,master - 0 0 5 connected
ad3a32dea36079f2620a3c83b2cad46ad34f458a 192.168.168.132:6394 master - 0 1640334460671 3 connected
fa735c03c464753311abeb12b148390c715b09ad 192.168.168.132:6392 master - 0 1640334456639 0 connected
c335d8defa53ab7595d1b191feb575341f925240 192.168.168.132:6390 master - 0 1640334459662 1 connected
f7922da8be6327bf979edc539c309e30d992d2a9 192.168.168.132:6391 master - 0 1640334458654 2 connected

[root@localhost redis_6390]# ss -anp|grep 54155
tcp LISTEN 0 128 192.168.168.132:16390 : users:((“redis-server”,pid=54155,fd=6))
tcp LISTEN 0 128 192.168.168.132:6390 : users:((“redis-server”,pid=54155,fd=4))
注意:集群中的每一个节点都会单独开辟一个 Tcp 通道,用于节点之间彼此通信,通信端口在基础端口上加10000

手动分配槽位

查看槽位分配情况,现在显示是失败

[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6390
192.168.168.132:6390> CLUSTER INFO
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_sent:2254
cluster_stats_messages_received:2254

分配槽位

[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6390 cluster addslots {0..5461}
OK
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6392 cluster addslots {5462..10922}
OK
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6394 cluster addslots {10923..16383}
OK
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6390 CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_sent:2940
cluster_stats_messages_received:2940

创建主从复制关系

登录从节点输入主节点的信息

[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6391 CLUSTER NODES
f7922da8be6327bf979edc539c309e30d992d2a9 192.168.168.132:6391 myself,master - 0 0 2 connected
ad3a32dea36079f2620a3c83b2cad46ad34f458a 192.168.168.132:6394 master - 0 1640336291181 3 connected 10923-16383
9a935f9f3ab69813c7521e3d0300534b8e702138 192.168.168.132:6395 master - 0 1640336289020 5 connected
fa735c03c464753311abeb12b148390c715b09ad 192.168.168.132:6392 master - 0 1640336290096 0 connected 5462-10922
c335d8defa53ab7595d1b191feb575341f925240 192.168.168.132:6390 master - 0 1640336292221 1 connected 0-5461
bee33ae4cadb6d222bb0cef4c58474aa8323124a 192.168.168.132:6393 master - 0 1640336293244 4 connected
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6391 CLUSTER REPLICATE fa735c03c464753311abeb12b148390c715b09ad
OK
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6393 CLUSTER REPLICATE ad3a32dea36079f2620a3c83b2cad46ad34f458a
OK
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6395 CLUSTER REPLICATE c335d8defa53ab7595d1b191feb575341f925240
OK
[root@localhost redis_6390]# redis-cli -h 192.168.168.132 -p 6390 CLUSTER NODES
c335d8defa53ab7595d1b191feb575341f925240 192.168.168.132:6390 myself,master - 0 0 1 connected 0-5461
fa735c03c464753311abeb12b148390c715b09ad 192.168.168.132:6392 master - 0 1640336404272 0 connected 5462-10922
ad3a32dea36079f2620a3c83b2cad46ad34f458a 192.168.168.132:6394 master - 0 1640336407298 3 connected 10923-16383
9a935f9f3ab69813c7521e3d0300534b8e702138 192.168.168.132:6395 slave c335d8defa53ab7595d1b191feb575341f925240 0 1640336405280 5 connected
bee33ae4cadb6d222bb0cef4c58474aa8323124a 192.168.168.132:6393 slave ad3a32dea36079f2620a3c83b2cad46ad34f458a 0 1640336403264 4 connected
f7922da8be6327bf979edc539c309e30d992d2a9 192.168.168.132:6391 slave fa735c03c464753311abeb12b148390c715b09ad 0 1640336406290 2 connected

ASK路由

当插入数据时,会发生如下现象

[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6390
192.168.168.132:6390> set k1 v1
(error) MOVED 12706 192.168.168.132:6394
192.168.168.132:6390> quit
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6394
192.168.168.132:6394> set k1 v1
OK
192.168.168.132:6394> quit

原因如下:
当用户发出指令,集群会随机分配槽位,如果槽位不在本机,必须登录相应的节点执行才行。
在这里插入图片描述

解决方法
在登录主机是加入-c参数

[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6390 -c
192.168.168.132:6390> set k4 v4
-> Redirected to slot [8455] located at 192.168.168.132:6392
OK

原理图:
当用户发出指令,集群会随机分配槽位,如果槽位不在本机,会自动请求目标主机,目标主机返回给用户
在这里插入图片描述

使用工具搭建部署Redis Cluster

手动搭建集群便于理解集群创建的流程和细节,不过手动搭建集群需要很多步骤,当集群节点众多时,必然会加大搭建集群的复杂度和运维成本,因此官方提供了 redis-trib.rb的工具方便我们快速搭建集群。
redis-trib.rb是采用 Ruby 实现的 redis 集群管理工具,内部通过 Cluster相关命令帮我们简化集群创建、检查、槽迁移和均衡等常见运维操作,使用前要安装 ruby 依赖环境

  1. 清除原本数据目录
[root@localhost redis_cluster]# rm -f redis_6390/*
[root@localhost redis_cluster]# rm -f redis_6391/*
[root@localhost redis_cluster]# rm -f redis_6392/*
[root@localhost redis_cluster]# rm -f redis_6393/*
[root@localhost redis_cluster]# rm -f redis_6394/*
[root@localhost redis_cluster]# rm -f redis_6395/*
[root@localhost redis_cluster]# pwd
/data/redis_cluster
[root@localhost redis_cluster]# redis-cli -h 192.168.168.132 -p 6390 shutdown
[root@localhost redis_cluster]# redis-cli -h 192.168.168.132 -p 6391 shutdown
[root@localhost redis_cluster]# redis-cli -h 192.168.168.132 -p 6392 shutdown
[root@localhost redis_cluster]# redis-cli -h 192.168.168.132 -p 6393 shutdown
[root@localhost redis_cluster]# redis-cli -h 192.168.168.132 -p 6394 shutdown
[root@localhost redis_cluster]# redis-cli -h 192.168.168.132 -p 6395 shutdown
  1. 安装命令
yum makecache fast
yum install rubygems -y
gem sources -a http://mirrors.aliyun.com/rubygems/
gem update –system
gem install redis -v 3.3.5
  1. 启动所有节点
[root@localhost mnt]# redis-server /opt/redis_cluster/redis_6390/conf/redis_cluster.conf 
[root@localhost mnt]# redis-server /opt/redis_cluster/redis_6391/conf/redis_cluster.conf 
[root@localhost mnt]# redis-server /opt/redis_cluster/redis_6392/conf/redis_cluster.conf 
[root@localhost mnt]# redis-server /opt/redis_cluster/redis_6393/conf/redis_cluster.conf 
[root@localhost mnt]# redis-server /opt/redis_cluster/redis_6394/conf/redis_cluster.conf 
[root@localhost mnt]# redis-server /opt/redis_cluster/redis_6395/conf/redis_cluster.conf
  1. 执行创建集群命令
[root@localhost src]# cd /opt/redis_cluster/redis/src
[root@localhost src]# ./redis-trib.rb create --replicas 1 192.168.168.132:6390 192.168.168.132:6392 192.168.168.132:6394 192.168.168.132:6395 192.168.168.132:6391 192.168.168.132:6393
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.168.132:6390
192.168.168.132:6392
192.168.168.132:6394
Adding replica 192.168.168.132:6395 to 192.168.168.132:6390
Adding replica 192.168.168.132:6391 to 192.168.168.132:6392
Adding replica 192.168.168.132:6393 to 192.168.168.132:6394
M: 2787e59294fe77836cea2aaa845390e67a6905f6 192.168.168.132:6390
   slots:0-5460 (5461 slots) master
M: 7d3d0ea74b32c9467994b2658cc46a7a9f7da1c3 192.168.168.132:6392
   slots:5461-10922 (5462 slots) master
M: 35ce4b12a3b8467a20f58ad95de64f461fdd0809 192.168.168.132:6394
   slots:10923-16383 (5461 slots) master
S: 3c51ec3ed4ba0a30b4fb3652835c8fb37f983fca 192.168.168.132:6395
   replicates 2787e59294fe77836cea2aaa845390e67a6905f6
S: 4dc371b8344ee6ca2d050494c6c2cf83893c713d 192.168.168.132:6391
   replicates 7d3d0ea74b32c9467994b2658cc46a7a9f7da1c3
S: 0e3bdca786a9f36f3621d7339d34b3d0b0ee0df8 192.168.168.132:6393
   replicates 35ce4b12a3b8467a20f58ad95de64f461fdd0809
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 192.168.168.132:6390)
M: 2787e59294fe77836cea2aaa845390e67a6905f6 192.168.168.132:6390
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 35ce4b12a3b8467a20f58ad95de64f461fdd0809 192.168.168.132:6394
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 4dc371b8344ee6ca2d050494c6c2cf83893c713d 192.168.168.132:6391
   slots: (0 slots) slave
   replicates 7d3d0ea74b32c9467994b2658cc46a7a9f7da1c3
S: 0e3bdca786a9f36f3621d7339d34b3d0b0ee0df8 192.168.168.132:6393
   slots: (0 slots) slave
   replicates 35ce4b12a3b8467a20f58ad95de64f461fdd0809
M: 7d3d0ea74b32c9467994b2658cc46a7a9f7da1c3 192.168.168.132:6392
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 3c51ec3ed4ba0a30b4fb3652835c8fb37f983fca 192.168.168.132:6395
   slots: (0 slots) slave
   replicates 2787e59294fe77836cea2aaa845390e67a6905f6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
#--replicas 1是每个节点都有一个复制节点,前三个是master节点,后面三个是对应的复制节点。但是会有一个问题,他不会按照上面的优秀拓扑图分配复制节点,自己还需要手动调整下,操作见创建主从复制关系
  1. 检查集群完整性
[root@localhost src]# ./redis-trib.rb check 192.168.168.132:6390

扩容节点

  1. 创建两个新节点
[root@localhost ~]# mkdir /data/redis_cluster/redis_6396
[root@localhost ~]# mkdir /data/redis_cluster/redis_6397
[root@localhost data]# cd /opt/redis_cluster/
[root@localhost redis_cluster]# cp -r redis_6395 redis_6396
[root@localhost redis_cluster]# cp -r redis_6395 redis_6397
[root@localhost redis_cluster]# sed -i 's/6395/6396/g' redis_6396/conf/redis_cluster.conf 
[root@localhost redis_cluster]# sed -i 's/6395/6397/g' redis_6397/conf/redis_cluster.conf
[root@localhost ~]# redis-server /opt/redis_cluster/redis_6396/conf/redis_cluster.conf 
[root@localhost ~]# redis-server /opt/redis_cluster/redis_6397/conf/redis_cluster.conf 
[root@localhost ~]# ps aux|grep redis
root      26006  0.0  0.1 139020  3032 ?        Ssl  21:43   0:01 redis-server 192.168.168.132:6390 [cluster]
root      26010  0.0  0.1 136972  2828 ?        Ssl  21:43   0:01 redis-server 192.168.168.132:6391 [cluster]
root      26014  0.0  0.1 139020  2960 ?        Ssl  21:43   0:01 redis-server 192.168.168.132:6392 [cluster]
root      26018  0.0  0.1 136972  2836 ?        Ssl  21:44   0:01 redis-server 192.168.168.132:6393 [cluster]
root      26022  0.0  0.1 139020  3064 ?        Ssl  21:44   0:01 redis-server 192.168.168.132:6394 [cluster]
root      26026  0.0  0.1 136972  2816 ?        Ssl  21:44   0:01 redis-server 192.168.168.132:6395 [cluster]
root      26319  0.0  0.1 136972  2500 ?        Ssl  22:18   0:00 redis-server 192.168.168.132:6396 [cluster]
root      26323  0.0  0.1 136972  2504 ?        Ssl  22:18   0:00 redis-server 192.168.168.132:6397 [cluster]
  1. 发现节点
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6390 cluster meet 192.168.168.132 6396
OK
[root@localhost ~]# redis-cli -h 192.168.168.132 -p 6390 cluster meet 192.168.168.132 6397
OK
  1. 使用工具扩容
[root@localhost src]# ./redis-trib.rb reshard 192.168.168.132:6390

输入每个节点分几个槽位
How many slots do you want to move (from 1 to 16384)?
输入要添加的master节点
What is the receiving node ID? xxxxxxxxx
输入每个主节点ID,或者直接输入all
ource node #1:all

  1. 将6397配置成备份节点
[root@localhost src]# redis-cli -h 192.168.168.132 -p 6397 CLUSTER REPLICATE 7ace1f9548a8e685dc7b2c36f870949db47a266a
OK
[root@localhost src]# redis-cli -h 192.168.168.132 -p 6390
192.168.168.132:6390> CLUSTER NODES
7ace1f9548a8e685dc7b2c36f870949db47a266a 192.168.168.132:6396 master - 0 1640356181381 8 connected 0-1364 5461-6826 10923-12287
35ce4b12a3b8467a20f58ad95de64f461fdd0809 192.168.168.132:6394 master - 0 1640356184498 3 connected 12288-16383
66e5775605f2e7fd9fca5fd6cc8237489dd0c325 192.168.168.132:6397 slave 7ace1f9548a8e685dc7b2c36f870949db47a266a 0 1640356179332 8 connected
4dc371b8344ee6ca2d050494c6c2cf83893c713d 192.168.168.132:6391 slave 7d3d0ea74b32c9467994b2658cc46a7a9f7da1c3 0 1640356183489 5 connected
2787e59294fe77836cea2aaa845390e67a6905f6 192.168.168.132:6390 myself,master - 0 0 1 connected 1365-5460
0e3bdca786a9f36f3621d7339d34b3d0b0ee0df8 192.168.168.132:6393 slave 35ce4b12a3b8467a20f58ad95de64f461fdd0809 0 1640356185570 6 connected
7d3d0ea74b32c9467994b2658cc46a7a9f7da1c3 192.168.168.132:6392 master - 0 1640356185873 2 connected 6827-10922
3c51ec3ed4ba0a30b4fb3652835c8fb37f983fca 192.168.168.132:6395 slave 2787e59294fe77836cea2aaa845390e67a6905f6 0 1640356182413 4 connected
192.168.168.132:6390>

收缩节点

  1. 将6496节点上的槽位分配到其他节点上
[root@localhost src]# ./redis-trib.rb reshard 192.168.168.132:6390
#这里填写需要转移的槽位,计算方法是16384/其他在线的主节点减去上次计算的4096.如果分配到最后一个节点看上面的信息看剩下多少槽位
#如下信息:
#M: 7ace1f9548a8e685dc7b2c36f870949db47a266a 192.168.168.132:6396
#   slots:6826,10923-12287 (1366 slots) master
How many slots do you want to move (from 1 to 16384)? 1365
#填写转移数据的主节点ID这里只能填写一个,所以得执行三次
What is the receiving node ID? 2787e59294fe77836cea2aaa845390e67a6905f6
#这里是填写需要转移数据得节点,及当前节点ID
Source node #1:7ace1f9548a8e685dc7b2c36f870949db47a266a
Source node #2:done
  1. 忘记节点
[root@localhost src]# ./redis-trib.rb del-node 192.168.168.132:6396 7ace1f9548a8e685dc7b2c36f870949db47a266a
>>> Removing node 7ace1f9548a8e685dc7b2c36f870949db47a266a from cluster 192.168.168.132:6396
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@localhost src]# ./redis-trib.rb del-node 192.168.168.132:6397 66e5775605f2e7fd9fca5fd6cc8237489dd0c325
>>> Removing node 66e5775605f2e7fd9fca5fd6cc8237489dd0c325 from cluster 192.168.168.132:6397
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

数据导入工具

  1. 安装工具
cd /opt/redis_cluster/
git clone https://github.com/vipshop/redis-migrate-tool.git
cd redis-migrate-tool/
autoreconf -fvi
./configure
make && make install 
  1. 创建配置文件
[root@localhost redis-migrate-tool]# vim redis_6379_to_6390.conf
[source]
type: single
servers:
- 192.168.168.132:6379
 
[target]
type: redis cluster
servers:
- 192.168.168.132:6390 
 
[common]
listen: 0.0.0.0:8888
source_safe: true
  1. 执行导入命令
[root@db01 ~]# redis-migrate-tool -c redis_6379_to_6390.conf 
  1. 数据校验
[root@db01 ~]# redis-migrate-tool -c redis_6379_to_6390.conf -C redis_check

分析键值大小

  1. 安装工具
yum install python-pip gcc python-devel 
cd /opt/
git clone https://github.com/sripathikrishnan/redis-rdb-tools
cd redis-rdb-tools
python setup.py install
  1. 使用
cd /data/redis_cluster/redis_6390/
rdb -c memory redis_6390.rdb -f redis_6390.rdb.csv
  1. 分析
awk -F ',' '{print $4,$2,$3,$1}' redis_6390.rdb.csv |sort  > 6390.txt
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值