Redis Cluster 重启某一节点

NealCaffrey_

已于 2023-07-20 18:08:10 修改

阅读量871

点赞数 1

文章标签： redis

于 2023-07-20 18:04:08 首次发布

本文链接：https://blog.csdn.net/siyanyanyanyai/article/details/131836483

版权

title: redis cluster 某节点手动重启
date: 2023-07-20
Auther: Neal
tags:

Redis

清单信息

IP	Port	ConfDir	DataDir
192.168.0.99	6379	/db/redis/conf/redis6379.conf	/db/redis/redis6379/data
192.168.0.99	6380	/db/redis/conf/redis6380.conf	/db/redis/redis6380/data
192.168.0.99	6381	/db/redis/conf/redis6381.conf	/db/redis/redis6381/data
192.168.0.99	6382	/db/redis/conf/redis6382.conf	/db/redis/redis6382/data
192.168.0.99	6383	/db/redis/conf/redis6383.conf	/db/redis/redis6383/data
192.168.0.99	6384	/db/redis/conf/redis6384.conf	/db/redis/redis6384/data

备份当前节点配置

# cd /db/redis/bin
# redis-cli -p 6379 -a 'xxxxxxxx'
> config rewrite
OK
> exit
# cp /db/redis/etc/redis.conf /db/redis/etc/redis.conf.20230720bak

确认集群状态

# cd /db/redis/bin
# redis-cli -p 6379 -a 'xxxxxxxx'
> cluster info

需要关注cluster_state状态为ok
示例输出：

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:7
cluster_my_epoch:7
cluster_stats_messages_ping_sent:4386
cluster_stats_messages_pong_sent:4407
cluster_stats_messages_auth-ack_sent:1
cluster_stats_messages_sent:8794
cluster_stats_messages_ping_received:4397
cluster_stats_messages_pong_received:4387
cluster_stats_messages_meet_received:10
cluster_stats_messages_auth-req_received:1
cluster_stats_messages_mfstart_received:1
cluster_stats_messages_received:8796

确认集群节点信息

执行IP和用户：10.3.3.93（root）

# cd /db/redis/bin
# redis-cli  -p 6379 -a 'xxxxxxxx'
> cluster nodes

确认当前主从角色比例为1:1。

示例输出

127.0.0.1:6379> CLUSTER NODES
174b43db772ae2e495d855eaf845f59e9229d854 192.168.0.99:6379@16379 myself,slave 5054bfc8845afa56fa537f60fe8a6b6730dcc6f7 0 1689755895000 1 connected
76a9840d2fe17068c4536a0df9eaa7f6d0e459af 192.168.0.99:6380@16380 master - 0 1689755896000 2 connected 5461-10922
168573e8961af11cf0a3922c77b1e29186710f53 192.168.0.99:6384@16384 slave 1009ad0a52f3930f97f169a57fbeff4ca037e2f0 0 1689755898144 6 connected
1009ad0a52f3930f97f169a57fbeff4ca037e2f0 192.168.0.99:6381@16381 master - 0 1689755896956 3 connected 10923-16383
5054bfc8845afa56fa537f60fe8a6b6730dcc6f7 192.168.0.99:6382@16382 master - 0 1689755895761 7 connected 0-5460
e50224353af5b0388d715fbbdc6df99edd180e10 192.168.0.99:6383@16383 slave 76a9840d2fe17068c4536a0df9eaa7f6d0e459af 0 1689755896000 5 connected

确认主从复制信息

# cd /db/redis/bin
# redis-cli -p 6381 -a 'xxxxxxxx'
> info replication

确认当前角色为master,slave的主机为10.3.3.94。

127.0.0.1:6381> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.0.99,port=6384,state=online,offset=6384,lag=0
master_replid:4eab3e8bc07b68d3287308105926789ff8bbe6bc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:6384
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:6384

生成redis快照文件

# cd /db/redis/bin
# redis-cli -p 6379 -a 'xxxxxxxx'
> BGSAVE

确认日志输出

4733:M 19 Jul 2023 15:14:02.092 * Starting BGSAVE for SYNC with target: disk
4733:M 19 Jul 2023 15:14:02.092 * Background saving started by pid 4850
4850:C 19 Jul 2023 15:14:02.096 * DB saved on disk
4850:C 19 Jul 2023 15:14:02.096 * RDB: 0 MB of memory used by copy-on-write
4733:M 19 Jul 2023 15:14:02.138 * Background saving terminated with success

创建AOF文件

# cd /db/redis/bin
# redis-cli -p 6379 -a 'xxxxxxxx'
> BGREWRITEAOF

日志示例输出

4733:M 19 Jul 2023 15:16:52.738 * Background saving started by pid 4868
4868:C 19 Jul 2023 15:16:52.739 * DB saved on disk
4868:C 19 Jul 2023 15:16:52.739 * RDB: 0 MB of memory used by copy-on-write
4733:M 19 Jul 2023 15:16:52.847 * Background saving terminated with success
4733:M 19 Jul 2023 15:17:03.881 * Background append only file rewriting started by pid 4877
4733:M 19 Jul 2023 15:17:04.376 * AOF rewrite child asks to stop sending diffs.
4877:C 19 Jul 2023 15:17:04.377 * Parent agreed to stop sending diffs. Finalizing AOF...
4877:C 19 Jul 2023 15:17:04.377 * Concatenating 0.00 MB of AOF diff received from parent.
4877:C 19 Jul 2023 15:17:04.377 * SYNC append only file rewrite performed
4877:C 19 Jul 2023 15:17:04.377 * AOF rewrite: 0 MB of memory used by copy-on-write
4733:M 19 Jul 2023 15:17:04.475 * Background AOF rewrite terminated with success

切换主从角色

登录的redis server为 192.168.0.99:6384的从节点
有的时候在主节点没有任何问题的情况下强制手动故障转移也是很有必要的，比如想要升级主节点的Redis进程，我们可以通过故障转移将其转为slave再进行升级操作来避免对集群的可用性造成很大的影响。Redis集群使用 CLUSTER FAILOVER命令来进行故障转移，不过要被转移的主节点的从节点上执行该命令手动故障转移比主节点失败自动故障转移更加安全，因为手动故障转移时客户端的切换是在确保新的主节点完全复制了失败的旧的主节点数据的前提下下发生的，所以避免了数据的丢失。

# cd /db/redis/bin
# redis-cli -p 6384 -a 'xxxxxxxx'
> cluster failover
> info replication
> cluster nodes
> cluster info

示例输出

127.0.0.1:6380> cluster failover
OK
127.0.0.1:6380> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.0.99,port=6383,state=online,offset=7462,lag=0
master_replid:52c2df692fd824aab498feae1adc61422b424037
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:7462
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:7462
127.0.0.1:6380> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:7
cluster_my_epoch:2
cluster_stats_messages_ping_sent:5523
cluster_stats_messages_pong_sent:5557
cluster_stats_messages_meet_sent:7
cluster_stats_messages_auth-ack_sent:1
cluster_stats_messages_sent:11088
cluster_stats_messages_ping_received:5554
cluster_stats_messages_pong_received:5531
cluster_stats_messages_meet_received:3
cluster_stats_messages_auth-req_received:1
cluster_stats_messages_received:11089
127.0.0.1:6380> cluster nodes
174b43db772ae2e495d855eaf845f59e9229d854 192.168.0.99:6379@16379 slave 5054bfc8845afa56fa537f60fe8a6b6730dcc6f7 0 1689756983310 7 connected
168573e8961af11cf0a3922c77b1e29186710f53 192.168.0.99:6384@16384 slave 1009ad0a52f3930f97f169a57fbeff4ca037e2f0 0 1689756984481 6 connected
76a9840d2fe17068c4536a0df9eaa7f6d0e459af 192.168.0.99:6380@16380 myself,master - 0 1689756980000 2 connected 5461-10922
e50224353af5b0388d715fbbdc6df99edd180e10 192.168.0.99:6383@16383 slave 76a9840d2fe17068c4536a0df9eaa7f6d0e459af 0 1689756981000 5 connected
5054bfc8845afa56fa537f60fe8a6b6730dcc6f7 192.168.0.99:6382@16382 master - 0 1689756982000 7 connected 0-5460
1009ad0a52f3930f97f169a57fbeff4ca037e2f0 192.168.0.99:6381@16381 master - 0 1689756982186 3 connected 10923-16383

停止redis server

# cd /db/redis/bin
# redis-cli -p 6379 -a 'xxxxxxxx'
> shutdown save
> exit

备份AOF文件

确认AOF文件位置

 cat /db/redis/conf/redis6379.conf  |grep -Ev '^#|^$'|grep dir

示例输出

dir "/db/redis/redis6379/data"

备份AOF文件

# cd /tmp/
# tar -czvf data.tar.gz `cat /db/redis/conf/redis6379.conf  |grep -Ev '^#|^$'|grep dir|awk -F '"' '{print $2}'`

示例输出

[root@redis tmp]# tar -czvf data.tar.gz `cat /db/redis/conf/redis6379.conf  |grep -Ev '^#|^$'|grep dir|awk -F '"' '{print $2}'`              
tar: Removing leading `/' from member names
/db/redis/redis6379/data/
/db/redis/redis6379/data/dump.rdb
/db/redis/redis6379/data/nodes.conf
/db/redis/redis6379/data/appendonly.aof
[root@redis tmp]#

检查备份文件

# ls /tmp/data.tar.gz

关闭主机

启动主机

启动redis server

# cd /db/redis/bin
# ./redis-server /db/redis/conf/redis6379.conf
# netstat -tunlp|grep 6379

示例输出

# netstat -tunlp|grep 6379
tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      5794/redis-server 0 
tcp        0      0 0.0.0.0:16379           0.0.0.0:*               LISTEN      5794/redis-server 0

确认集群状态

# cd /db/redis/bin
# redis-cli  -p 6379 -a 'xxxxxxxx'
> info replication
> cluster nodes
> cluster info

示例输出

127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.0.99
master_port:6382
master_link_status:up
master_last_io_seconds_ago:4
master_sync_in_progress:0
slave_repl_offset:8036
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:b993866b091f99ff92d4271c58d125bcb29b75d8
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:8036
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:7771
repl_backlog_histlen:266
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:9
cluster_my_epoch:9
cluster_stats_messages_ping_sent:198
cluster_stats_messages_pong_sent:190
cluster_stats_messages_sent:388
cluster_stats_messages_ping_received:190
cluster_stats_messages_pong_received:198
cluster_stats_messages_received:388
127.0.0.1:6379> cluster nodes
5054bfc8845afa56fa537f60fe8a6b6730dcc6f7 192.168.0.99:6382@16382 master - 0 1689757489051 9 connected 0-5460
e50224353af5b0388d715fbbdc6df99edd180e10 192.168.0.99:6383@16383 slave 76a9840d2fe17068c4536a0df9eaa7f6d0e459af 0 1689757490205 5 connected
76a9840d2fe17068c4536a0df9eaa7f6d0e459af 192.168.0.99:6380@16380 master - 0 1689757488000 2 connected 5461-10922
174b43db772ae2e495d855eaf845f59e9229d854 192.168.0.99:6379@16379 myself,slave 5054bfc8845afa56fa537f60fe8a6b6730dcc6f7 0 1689757489000 8 connected
1009ad0a52f3930f97f169a57fbeff4ca037e2f0 192.168.0.99:6381@16381 master - 0 1689757487000 3 connected 10923-16383
168573e8961af11cf0a3922c77b1e29186710f53 192.168.0.99:6384@16384 slave 1009ad0a52f3930f97f169a57fbeff4ca037e2f0 0 1689757487000 6 connected

复制延迟确认

info replication讲解

主节点

127.0.0.1:6379> info replication
# Replication
# 角色
role:master
# 从节点的连接数
connected_slaves:2
# 从节点详细信息 IP PORT 状态 命令(单位:字节长度)偏移量 延迟秒数
# 主节点每次处理完写操作，会把命令的字节长度累加到master_repl_offset中。
# 从节点在接收到主节点发送的命令后，会累加记录子什么偏移量信息slave_repl_offset，同时，也会每秒钟上报自身的复制偏移量到主节点，以供主节点记录存储。
# 在实际应用中，可以通过对比主从复制偏移量信息来监控主从复制健康状况。
slave0:ip=192.168.10.102,port=6379,state=online,offset=23866,lag=0
slave1:ip=192.168.10.103,port=6379,state=online,offset=23866,lag=0
# master启动时生成的40位16进制的随机字符串，用来标识master节点
master_replid:acc2aaa1f0bb0fd79d7d3302f16bddcbe4add423
master_replid2:0000000000000000000000000000000000000000
# master 命令(单位:字节长度)已写入的偏移量
master_repl_offset:23866
second_repl_offset:-1
# 0/1：关闭/开启复制积压缓冲区标志(2.8+)，主要用于增量复制及丢失命令补救
repl_backlog_active:1
# 缓冲区最大长度，默认 1M
repl_backlog_size:1048576
# 缓冲区起始偏移量
repl_backlog_first_byte_offset:1
# 缓冲区已存储的数据长度
repl_backlog_histlen:23866

从节点

127.0.0.1:6379> info replication
# Replication
# 角色
role:slave
# 主节点详细信息
master_host:192.168.10.101
master_port:6379
# slave端可查看它与master之间同步状态,当复制断开后表示down
master_link_status:up
# 主库多少秒未发送数据到从库
master_last_io_seconds_ago:1
# 从服务器是否在与主服务器进行同步 0否/1是
master_sync_in_progress:0
# slave复制命令(单位:字节长度)偏移量
slave_repl_offset:24076
# 选举时，成为主节点的优先级，数字越大优先级越高，0 永远不会成为主节点

slave_priority:100
# 从库是否设置只读，0读写/1只读
slave_read_only:1
# 连接的slave实例个数
connected_slaves:0
# master启动时生成的40位16进制的随机字符串，用来标识master节点
master_replid:acc2aaa1f0bb0fd79d7d3302f16bddcbe4add423
# slave切换master之后，会生成了自己的master标识，之前的master节点的标识存到了master_replid2的位置
master_replid2:0000000000000000000000000000000000000000
# master 命令(单位:字节长度)已写入的偏移量
master_repl_offset:24076
# 主从切换时记录主节点的命令偏移量+1，为了避免全量复制
second_repl_offset:-1
# 0/1：关闭/开启复制积压缓冲区标志(2.8+)，主要用于增量复制及丢失命令补救
repl_backlog_active:1
# 缓冲区最大长度，默认 1M
repl_backlog_size:1048576
# 缓冲区起始偏移量
repl_backlog_first_byte_offset:1
# 缓冲区已存储的数据长度
repl_backlog_histlen:24076

日志查看

tail -f -n 1000 /usr/local/redis/log/redis.log
# 准备就绪，接受客户端连接
* Ready to accept connections
# 102 从节点发起 SYNC 请求
* Replica 192.168.10.102:6379 asks for synchronization
# 全量
# 从节点发起全量复制请求
* Full resync requested by replica 192.168.10.102:6379
# 创建 repl_backlog 文件及生成 master_replid
* Replication backlog created, my new replication IDs are 'acc2aaa1f0bb0fd79d7d3302f16bddcbe4add423' and '0000000000000000000000000000000000000000'
# 通过 BGSAVE 指令将数据写入磁盘(RBD操作)
* Starting BGSAVE for SYNC with target: disk
# 开启一个子守护进程执行写入
* Background saving started by pid 1377
# 数据已写入磁盘
* DB saved on disk
# 有 4MB 数据已写入磁盘
* RDB: 4 MB of memory used by copy-on-write
# 保存结束
* Background saving terminated with success
# 从节点同步数据结束
* Synchronization with replica 192.168.10.102:6379 succeeded
# 103 从节点发起 SYNC 请求，执行同步数据操作
* Replica 192.168.10.103:6379 asks for synchronization
# 从节点发起全量复制请求
* Full resync requested by replica 192.168.10.103:6379

# 增量
# 当前一个客户端连接，执行了两个复制
1 clients connected (2 replicas), 1955144 bytes in use

重启节点加入集群异常

出现重启后此节点无法自动匹配主从关机或丢失数据后，可参照入下步骤进行重新加入

--cluster fix 此命令谨慎执行，此方案中未涉及

确认当前集群信息

# redis-cli  -a 'xxxxxxxx' -p 6382
127.0.0.1:6382> cluster nodes
e50224353af5b0388d715fbbdc6df99edd180e10 192.168.0.99:6383@16383 slave 76a9840d2fe17068c4536a0df9eaa7f6d0e459af 0 1689773132000 5 connected
1009ad0a52f3930f97f169a57fbeff4ca037e2f0 192.168.0.99:6381@16381 master - 0 1689773132764 3 connected 10923-16383
168573e8961af11cf0a3922c77b1e29186710f53 192.168.0.99:6384@16384 slave 1009ad0a52f3930f97f169a57fbeff4ca037e2f0 0 1689773133826 6 connected
174b43db772ae2e495d855eaf845f59e9229d854 192.168.0.99:6379@16379 master - 0 1689773131000 8 connected
76a9840d2fe17068c4536a0df9eaa7f6d0e459af 192.168.0.99:6380@16380 master - 0 1689773132000 2 connected 5461-10922
5054bfc8845afa56fa537f60fe8a6b6730dcc6f7 192.168.0.99:6382@16382 myself,master - 0 1689773129000 9 connected 0-5460

集群中剔除问题节点

cluster forget {node_id}中node_id为redis自动生成的每个节点的唯一标识。可从cluster nodes中查看

127.0.0.1:6382> cluster forget 174b43db772ae2e495d855eaf845f59e9229d854
OK

清理故障节点数据

重启前为slave角色，顾无需备份数据目录

127.0.0.1:6379> flushdb
(error) READONLY You can't write against a read only replica.
127.0.0.1:6379> cluster reset soft
OK
127.0.0.1:6379> exit

重新加入节点

--cluster-master-id 值为cluster failover后的新主ID
add-node 后机器清单为集群中一对主从的机器清单，前为重新加入的节点，后为对应的主节点

[root@redis ~]# redis-cli  -a 'xxxxxxxx' -p 6382 --cluster add-node  192.168.0.99:6379 192.168.0.99:6382 --cluster-slave --cluster-master-id 5054bfc8845afa56fa537f60fe8a6b6730dcc6f7

[OK] New node added correctly.