Redis Cluster 部署、在线扩容、缩容、迁移、集群管理实践

概述

Redis 分片集群是为了应对业务增长带来的数据增长, 实现在线动态横向扩缩容，集群中的每个分片主节点处理 16384 个哈希槽的一个子集; 为了稳定也可以进行高可用部署, 即将各分片节点启用一个或多个副本节点; 当分片主节点挂掉,其对应的副本节点自动提升为新主节点,当原主节点重新上线后,自动成为新主节点的副本节点; 当一对分片主从节点都故障时,整个集群不能正常工作，需要人工介入恢复启动故障节点

分片集群

1、创建一个有 3 个分片节点的集群

# 创建具有 3 个分片节点的集群、每个分片有 1 个副本

# redis-cli --cluster create 192.168.31.17:6384 192.168.31.18:6384 192.168.31.19:6384 --cluster-replicas 1

# 创建具有 3 个分片节点的集群, 每个分片默认无副本

redis-cli --cluster create 192.168.31.17:6384 192.168.31.18:6384 192.168.31.19:6384

> >>> Performing hash slots allocation on 3 nodes...

> Master[0] -> Slots 0 - 5460

> Master[1] -> Slots 5461 - 10922

> Master[2] -> Slots 10923 - 16383

> M: 1db08d1048b20b22dfbf5c5d724d95f46a843d29 192.168.31.17:6384

> slots:[0-5460] (5461 slots) master

> M: 3f9d82efdf04bb5fcf3741de4ea9314925bbeefd 192.168.31.18:6384

> slots:[5461-10922] (5462 slots) master

> M: 16a0c6947fcb7d498b7c96aa5b5a70491298e966 192.168.31.19:6384

> slots:[10923-16383] (5461 slots) master

> Can I set the above configuration? (type 'yes' to accept): yes

> >>> Nodes configuration updated

> >>> Assign a different config epoch to each node

> >>> Sending CLUSTER MEET messages to join the cluster

> Waiting for the cluster to join

> ..

> >>> Performing Cluster Check (using node 192.168.31.17:6384)

> M: 1db08d1048b20b22dfbf5c5d724d95f46a843d29 192.168.31.17:6384

> slots:[0-5460] (5461 slots) master

> M: 16a0c6947fcb7d498b7c96aa5b5a70491298e966 192.168.31.19:6384

> slots:[10923-16383] (5461 slots) master

> M: 3f9d82efdf04bb5fcf3741de4ea9314925bbeefd 192.168.31.18:6384

> slots:[5461-10922] (5462 slots) master

> [OK] All nodes agree about slots configuration.

> >>> Check for open slots...

> >>> Check slots coverage...

> [OK] All 16384 slots covered.

验证

1、查看指定 redis 节点状态(包括键数、内存使用量、请求数、连接数等)

redis-cli --stat -p 6384 -h 192.168.31.17

> ------- data ------ --------------------- load -------------------- - child -

> keys mem clients blocked requests connections

> 30022 9.25M 2 0 30518 (+2) 59

> 30022 9.25M 2 0 30520 (+2) 59

> 30022 9.25M 2 0 30522 (+2) 59

> 30022 9.25M 2 0 30524 (+2) 59

> 30022 9.25M 2 0 30526 (+2) 59

> 30022 9.25M 2 0 30528 (+2) 59

> 30022 9.25M 2 0 30530 (+2) 59

2、查看各主节点哈希槽、键的分布情况, 以及副本数、键总数


redis-cli --cluster info 192.168.31.17:6384

> 192.168.31.18:6384 (79c19006...) -> 49888 keys | 5462 slots | 1 slaves.

> 192.168.31.17:6385 (3faafe60...) -> 50115 keys | 5461 slots | 1 slaves.

> 192.168.31.19:6384 (df7d9f8e...) -> 49995 keys | 5461 slots | 1 slaves.

> [OK] 149998 keys in 3 masters.

> 9.16 keys per slot on average.

集群管理

1、redis-cli 交互式命令行

redis-cli -p 6384 -h 192.168.31.17

# 查看集群节点信息

cluster nodes

# meet 添加节点, 任意节点执行即可

cluster meet 192.168.31.17 6385

# forget 根据节点 ID 删除节点, 需要在所有节点执行该命令

cluster forget 82d14741e161d07012783ae7c9bf1ef20ddc28a0

# 将当前 cluster 节点配置为指定主节点(1db08d1048b20b22dfbf5c5d724d95f46a843d29)的副本节点

cluster replicate 1db08d1048b20b22dfbf5c5d724d95f46a843d29

# 查看指定 master 节点(1db08d1048b20b22dfbf5c5d724d95f46a843d29)的副本节点

cluster replicas 1db08d1048b20b22dfbf5c5d724d95f46a843d29

cluster reset # 重置当前节点集群状态文件 nodes.conf

2、redis-cli –cluster 子命令

# 在集群中所有节点上执行命令 client list

redis-cli --cluster call 192.168.31.17:6384 client list

# 将节点 192.168.31.17:6385 添加为新的主节点, 需要从其它主节点迁移部分槽位到该新主节点才能正常工作

redis-cli --cluster add-node 192.168.31.17:6385 192.168.31.17:6384

# 将节点 192.168.31.17:6385 添加为主节点(1db08d1048b20b22dfbf5c5d724d95f46a843d29)的从节点

redis-cli --cluster add-node 192.168.31.17:6385 192.168.31.17:6384 --cluster-slave --cluster-master-id 1db08d1048b20b22dfbf5c5d724d95f46a843d29

# 删除节点 82d14741e161d07012783ae7c9bf1ef20ddc28a0 且关闭该节点

redis-cli --cluster del-node 192.168.31.17:6384 82d14741e161d07012783ae7c9bf1ef20ddc28a0

# 从指定节点迁移 1024 个槽位到目标节点 073ad3a6dfe64289be66c6d8e302b29afc4b0ec7

redis-cli --cluster reshard 192.168.31.17:6385 --cluster-from 1db08d1048b20b22dfbf5c5d724d95f46a843d29 --cluster-to 073ad3a6dfe64289be66c6d8e302b29afc4b0ec7 --cluster-slots 1024

# 使用如下命令, 进入交互式输入对应参数值进行节点间的槽位迁移

redis-cli --cluster reshard 192.168.31.17:6384

# 当各节点槽位不均衡时,执行该命令、自动均衡各节点槽位 slots

redis-cli --cluster rebalance 192.168.31.17:6384

扩容

1、集群主节点扩容, 由 3 个主节点扩展到 5 个主节点

// 添加新节点 192.168.31.{18,19}:6386

redis-cli --cluster add-node 192.168.31.18:6386 192.168.31.17:6384

redis-cli --cluster add-node 192.168.31.19:6386 192.168.31.17:6384

// 自动平衡集群各主节点哈希槽的分布

redis-cli --cluster rebalance 192.168.31.17:6384 --cluster-use-empty-masters

redis-cli --cluster info 192.168.31.17:6384

> 192.168.31.18:6386 (c64357f2...) -> 29894 keys | 3278 slots | 0 slaves.

> 192.168.31.18:6384 (79c19006...) -> 29927 keys | 3276 slots | 1 slaves.

> 192.168.31.17:6385 (3faafe60...) -> 29835 keys | 3276 slots | 1 slaves.

> 192.168.31.19:6384 (df7d9f8e...) -> 30083 keys | 3276 slots | 1 slaves.

> 192.168.31.19:6386 (a0f71de7...) -> 30259 keys | 3278 slots | 0 slaves.

> [OK] 149998 keys in 5 masters.

> 9.16 keys per slot on average.

缩容

1、集群主节点扩容, 由 5 个主节点扩展到 3 个主节点

// 迁移节点 192.168.31.18:6386 的哈希槽

redis-cli --cluster reshard 192.168.31.17:6384 \

--cluster-from c64357f288902d1209fdf91f4804bf13b2a59fcf \

--cluster-to 79c19006ae3dcd12522090ddcc2b5b917b0feb2a \

--cluster-slots 3278 \

--cluster-yes

// 迁移节点 192.168.31.19:6386 的哈希槽

redis-cli --cluster reshard 192.168.31.17:6384 \

--cluster-from a0f71de78c2355f99de2b4c19796afb02e23c623 \

--cluster-to df7d9f8e2d7dbad37bdd1094c2770d1feabbd228 \

--cluster-slots 3278 \

--cluster-yes

// 查看各主节点哈希槽、键的分布情况

redis-cli --cluster info 192.168.31.17:6384

> 192.168.31.18:6386 (c64357f2...) -> 0 keys | 0 slots | 0 slaves.

> 192.168.31.18:6384 (79c19006...) -> 59821 keys | 6554 slots | 1 slaves.

> 192.168.31.17:6385 (3faafe60...) -> 29835 keys | 3276 slots | 1 slaves.

> 192.168.31.19:6384 (df7d9f8e...) -> 60342 keys | 6554 slots | 1 slaves.

> 192.168.31.19:6386 (a0f71de7...) -> 0 keys | 0 slots | 0 slaves.

> [OK] 149998 keys in 5 masters.

> 9.16 keys per slot on average.

// 自动平衡集群各主节点哈希槽的分布

redis-cli --cluster rebalance 192.168.31.17:6384

// 删除节点 192.168.31.{18,19}:6386

redis-cli --cluster del-node 192.168.31.17:6384 c64357f288902d1209fdf91f4804bf13b2a59fcf

redis-cli --cluster del-node 192.168.31.17:6384 a0f71de78c2355f99de2b4c19796afb02e23c623

// 查看各主节点哈希槽、键的分布情况

redis-cli --cluster info 192.168.31.17:6384

> 192.168.31.18:6384 (79c19006...) -> 49762 keys | 5461 slots | 1 slaves.

> 192.168.31.17:6385 (3faafe60...) -> 50060 keys | 5462 slots | 1 slaves.

> 192.168.31.19:6384 (df7d9f8e...) -> 50176 keys | 5461 slots | 1 slaves.

> [OK] 149998 keys in 3 masters.

> 9.16 keys per slot on average.

迁移

Redis-shake 是一个用于在两个 Redis 之间同步数据的工具,支持 Redis Cluster 的在线迁移与离线迁移(备份文件导入)
支持解析、恢复、备份、同步四个功能

1、在线迁移

首次全量同步,然后进行增量的拉取(通过 psync 命令)

# 创建迁移目标 redis 集群

redis-cli --cluster create 192.168.31.17:6386 192.168.31.18:6386 192.168.31.19:6386

# cat redis-shake.conf 源及目标端主要配置项, 其它配置项默认即可

source.type: cluster

# master 可改为 slave, 表示从副本同步数据到目标端, 192.168.31.19 为任意源端 redis 节点

source.address: master@192.168.31.19:6384

target.type: cluster

target.address: master@192.168.31.19:6386

./redis-shake -type sync -conf redis-shake.conf

2、离线迁移

适用于源实例与目标实例的网络无法连通的场景,或源端实例部署在其他云厂商 Redis 服务中,无法实现在线迁移

# cat redis-shake.conf 源及目标端主要配置项, 其它配置项默认即可

source.address: master@192.168.31.19:6384

# restore 导入数据时指定的备份文件

source.rdb.input: local_dump.0;local_dump.1;local_dump.2

target.type: cluster

target.address: master@192.168.31.19:6386

备份: ./redis-shake -type dump -conf redis-shake.conf

恢复: ./redis-shake -type restore -conf redis-shake.conf

来源：Redis Cluster 部署、在线扩容、缩容、迁移、集群管理实践-武穆逸仙https://www.iwmyx.cn/redisclusterbszxa.html

---

来源：运维技术帮