基于 Redis-5.0.5 创建 Simple Redis-Cluster

     目前Redis 依旧是比较流行的缓存中间件,开始接触Redis,大概是在四年前。当时只知道它是K-V 的内存数据库,一般作数据缓存用,也没有自己研究过。后来参与过其他项目,用到了分布式锁等功能。由于公司直接采购的阿里云Redis ,有丰富的监控 和 集群功能,所以一直没有机会深入的理解Redis 集群的运行机制,所以想通过开设专栏记录学习Redis 的整个过程。

    还是那句话, 看过千遍,不如自己动手来一遍。

一、手动配置创建

     redis-cluster 最少实例个数为6个,3主3从。如果使用的是Redis 5,这很容易实现,这是因为redis-cli中嵌入了Redis Cluster命令行实用程序,我们可以使用它来创建新集群,检查或重新分片现有集群等等。对于Redis版本3或4,有一个称为redis-trib.rb的较旧工具,和redis-cli cluster 功能相似。可以在Redis源代码分发的src目录中找到它。 需要安装redis gem才能运行redis-trib,以下使用5.0.5版本. 

    1、redis.conf 配置。其中nodes.conf 每个实例一个配置文件,如果在相同目录下,文件名称不能一样。

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

   2、分别创建6个目录,存放每个实例的redis.conf 配置文件

mkdir cluster-test
cd cluster-test
mkdir 7000 7001 7002 7003 7004 7005

  3、分别启动这6个实例

../redis-server ./redis.conf

实例启动时都会分配一个ID,这个ID 是不会变的,是为了在cluster context 中保证每个实例的唯一性,跟实例的host address 和 port 无关。该ID 保存在nodes.conf 文件中,集群重启时,会从nodes.conf 文件中加载实例信息及集群分布信息

1431:M 12 Apr 2020 10:21:05.117 * Node configuration loaded, I'm 272601ed435722164eec803da6a68d036ed133be

  4、创建集群

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 --cluster-replicas 1

 执行完创建集群的命令后,会提示: [OK] All 16384 slots covered.

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
Adding replica 127.0.0.1:7003 to 127.0.0.1:7002
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 272601ed435722164eec803da6a68d036ed133be 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
M: f0ddc47c9674761642125fe6ee813e113c4b4717 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
M: 85a281c31f28583f9257d59c56353445d633ff72 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
S: b2af5423c20d95bfcb29a92ea259a015d07ad4a8 127.0.0.1:7003
   replicates f0ddc47c9674761642125fe6ee813e113c4b4717
S: 19927db6574a0436be0e2b6850c26211f8800259 127.0.0.1:7004
   replicates 85a281c31f28583f9257d59c56353445d633ff72
S: fefd402843feba00ad875d10c878c478f5d44f80 127.0.0.1:7005
   replicates 272601ed435722164eec803da6a68d036ed133be
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.....
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 272601ed435722164eec803da6a68d036ed133be 127.0.0.1:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: b2af5423c20d95bfcb29a92ea259a015d07ad4a8 127.0.0.1:7003
   slots: (0 slots) slave
   replicates f0ddc47c9674761642125fe6ee813e113c4b4717
M: 85a281c31f28583f9257d59c56353445d633ff72 127.0.0.1:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: fefd402843feba00ad875d10c878c478f5d44f80 127.0.0.1:7005
   slots: (0 slots) slave
   replicates 272601ed435722164eec803da6a68d036ed133be
M: f0ddc47c9674761642125fe6ee813e113c4b4717 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 19927db6574a0436be0e2b6850c26211f8800259 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 85a281c31f28583f9257d59c56353445d633ff72
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

  主(7001) 与从(7003) 两实例之间通信交互: 

1432:M 12 Apr 2020 10:21:10.319 # Server initialized
1432:M 12 Apr 2020 10:21:10.319 * Ready to accept connections
1432:M 12 Apr 2020 10:22:22.162 # configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH
1432:M 12 Apr 2020 10:22:22.294 # IP address for this node updated to 127.0.0.1
1432:M 12 Apr 2020 10:22:27.184 # Cluster state changed: ok
1432:M 12 Apr 2020 10:22:28.627 * Replica 127.0.0.1:7003 asks for synchronization
1432:M 12 Apr 2020 10:22:28.627 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '87986b2d4f8cf885def3435c21cb8201334aa884', my replication IDs are '7e99a703ec7509f04dfdda283abc8376d31c498f' and '0000000000000000000000000000000000000000')
1432:M 12 Apr 2020 10:22:28.627 * Starting BGSAVE for SYNC with target: disk
1432:M 12 Apr 2020 10:22:28.627 * Background saving started by pid 1440
1440:C 12 Apr 2020 10:22:28.629 * DB saved on disk
1432:M 12 Apr 2020 10:22:28.728 * Background saving terminated with success
1432:M 12 Apr 2020 10:22:28.729 * Synchronization with replica 127.0.0.1:7003 succeeded
1434:M 12 Apr 2020 10:21:20.660 # Server initialized
1434:M 12 Apr 2020 10:21:20.660 * Ready to accept connections
1434:M 12 Apr 2020 10:22:22.163 # configEpoch set to 4 via CLUSTER SET-CONFIG-EPOCH
1434:M 12 Apr 2020 10:22:22.294 # IP address for this node updated to 127.0.0.1
1434:M 12 Apr 2020 10:22:27.289 # Cluster state changed: ok
1434:S 12 Apr 2020 10:22:28.204 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1434:S 12 Apr 2020 10:22:28.624 * Connecting to MASTER 127.0.0.1:7001
1434:S 12 Apr 2020 10:22:28.626 * MASTER <-> REPLICA sync started
1434:S 12 Apr 2020 10:22:28.626 * Non blocking connect for SYNC fired the event.
1434:S 12 Apr 2020 10:22:28.626 * Master replied to PING, replication can continue...
1434:S 12 Apr 2020 10:22:28.626 * Trying a partial resynchronization (request 87986b2d4f8cf885def3435c21cb8201334aa884:1).
1434:S 12 Apr 2020 10:22:28.628 * Full resync from master: 189eac65491ebe70567b23d82b1f3b310d505ede:0
1434:S 12 Apr 2020 10:22:28.628 * Discarding previously cached master state.
1434:S 12 Apr 2020 10:22:28.729 * MASTER <-> REPLICA sync: receiving 175 bytes from master
1434:S 12 Apr 2020 10:22:28.729 * MASTER <-> REPLICA sync: Flushing old data
1434:S 12 Apr 2020 10:22:28.729 * MASTER <-> REPLICA sync: Loading DB in memory
1434:S 12 Apr 2020 10:22:28.730 * MASTER <-> REPLICA sync: Finished with success
1434:S 12 Apr 2020 10:22:28.731 * Background append only file rewriting started by pid 1441
1434:S 12 Apr 2020 10:22:28.756 * AOF rewrite child asks to stop sending diffs.
1441:C 12 Apr 2020 10:22:28.756 * Parent agreed to stop sending diffs. Finalizing AOF...
1441:C 12 Apr 2020 10:22:28.757 * Concatenating 0.00 MB of AOF diff received from parent.
1441:C 12 Apr 2020 10:22:28.757 * SYNC append only file rewrite performed
1434:S 12 Apr 2020 10:22:28.832 * Background AOF rewrite terminated with success
1434:S 12 Apr 2020 10:22:28.832 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1434:S 12 Apr 2020 10:22:28.832 * Background AOF rewrite finished successfully

 7001 节点的nodes-1.conf 文件内容,可以看到,总共有6个节点,每个节点的ip和port,主从节点的对应关系,如下所示:

cat nodes-1.conf
f0ddc47c9674761642125fe6ee813e113c4b4717 127.0.0.1:7001@17001 myself,master - 0 1586658147000 2 connected 5461-10922
85a281c31f28583f9257d59c56353445d633ff72 127.0.0.1:7002@17002 master - 0 1586658148525 3 connected 10923-16383
19927db6574a0436be0e2b6850c26211f8800259 127.0.0.1:7004@17004 slave 85a281c31f28583f9257d59c56353445d633ff72 1586658149557 1586658147000 5 connected
b2af5423c20d95bfcb29a92ea259a015d07ad4a8 127.0.0.1:7003@17003 slave f0ddc47c9674761642125fe6ee813e113c4b4717 0 1586658148524 4 connected
fefd402843feba00ad875d10c878c478f5d44f80 127.0.0.1:7005@17005 slave 272601ed435722164eec803da6a68d036ed133be 0 1586658149558 6 connected
272601ed435722164eec803da6a68d036ed133be 127.0.0.1:7000@17000 master - 0 1586658149000 1 connected 0-5460

 5、简单的使用

./redis-cli -c -p 7000

127.0.0.1:7000> set name apple
-> Redirected to slot [5798] located at 127.0.0.1:7001
OK
127.0.0.1:7001> set age  18
-> Redirected to slot [741] located at 127.0.0.1:7000
OK

6、failover 

  ①、将主节点7001 停止运行,可以发现从节点经过6次sync后未成功,就会认为主节点fail,将该主节点在它的nodes-3.conf 节点配置中标识为fail,并同步主节点fail 状态到其他示例

1434:S 12 Apr 2020 11:50:43.481 # Connection with master lost.
1434:S 12 Apr 2020 11:50:43.481 * Caching the disconnected master state.
1434:S 12 Apr 2020 11:50:44.256 * Connecting to MASTER 127.0.0.1:7001
1434:S 12 Apr 2020 11:50:44.258 * MASTER <-> REPLICA sync started
1434:S 12 Apr 2020 11:50:44.259 # Error condition on socket for SYNC: Connection refused
1434:S 12 Apr 2020 11:50:45.304 * Connecting to MASTER 127.0.0.1:7001
1434:S 12 Apr 2020 11:50:45.307 * MASTER <-> REPLICA sync started
1434:S 12 Apr 2020 11:50:45.308 # Error condition on socket for SYNC: Connection refused
1434:S 12 Apr 2020 11:50:46.359 * Connecting to MASTER 127.0.0.1:7001
1434:S 12 Apr 2020 11:50:46.362 * MASTER <-> REPLICA sync started
1434:S 12 Apr 2020 11:50:46.363 # Error condition on socket for SYNC: Connection refused
1434:S 12 Apr 2020 11:50:47.411 * Connecting to MASTER 127.0.0.1:7001
1434:S 12 Apr 2020 11:50:47.412 * MASTER <-> REPLICA sync started
1434:S 12 Apr 2020 11:50:47.413 # Error condition on socket for SYNC: Connection refused
1434:S 12 Apr 2020 11:50:48.455 * Connecting to MASTER 127.0.0.1:7001
1434:S 12 Apr 2020 11:50:48.458 * MASTER <-> REPLICA sync started
1434:S 12 Apr 2020 11:50:48.459 # Error condition on socket for SYNC: Connection refused
1434:S 12 Apr 2020 11:50:49.509 * Connecting to MASTER 127.0.0.1:7001
1434:S 12 Apr 2020 11:50:49.512 * MASTER <-> REPLICA sync started
1434:S 12 Apr 2020 11:50:49.513 # Error condition on socket for SYNC: Connection refused
1434:S 12 Apr 2020 11:50:49.514 * FAIL message received from fefd402843feba00ad875d10c878c478f5d44f80 about f0ddc47c9674761642125fe6ee813e113c4b4717
1434:S 12 Apr 2020 11:50:49.514 # Cluster state changed: fail
1434:S 12 Apr 2020 11:50:49.618 # Start of election delayed for 907 milliseconds (rank #0, offset 7253).
1434:S 12 Apr 2020 11:50:50.549 * Connecting to MASTER 127.0.0.1:7001
1434:S 12 Apr 2020 11:50:50.551 * MASTER <-> REPLICA sync started
1434:S 12 Apr 2020 11:50:50.552 # Starting a failover election for epoch 8.
1434:S 12 Apr 2020 11:50:50.554 # Error condition on socket for SYNC: Connection refused
1434:S 12 Apr 2020 11:50:50.555 # Failover election won: I'm the new master.
1434:S 12 Apr 2020 11:50:50.555 # configEpoch set to 8 after successful failover
1434:M 12 Apr 2020 11:50:50.555 # Setting secondary replication ID to 189eac65491ebe70567b23d82b1f3b310d505ede, valid up to offset: 7254. New replication ID is 4e4387a4158d3302aa6fdd8a829e06913ad65879
1434:M 12 Apr 2020 11:50:50.555 * Discarding previously cached master state.
1434:M 12 Apr 2020 11:50:50.556 # Cluster state changed: ok

 从节点的 nodes-3.conf 内容如下,可以看到已经将主节点(后4位是4747)标记为fail,并且从节点已升级为主节点。虽然主节点已经fail了,但是 nodes-3.conf 文件中还保留它ID等信息,这是为了在主节点恢复运行后,重新加入到cluster中使用。 

fefd402843feba00ad875d10c878c478f5d44f80 127.0.0.1:7005@17005 master - 0 1586663449000 7 connected 0-5460
b2af5423c20d95bfcb29a92ea259a015d07ad4a8 127.0.0.1:7003@17003 myself,master - 0 1586663449000 8 connected 5461-10922
272601ed435722164eec803da6a68d036ed133be 127.0.0.1:7000@17000 slave fefd402843feba00ad875d10c878c478f5d44f80 0 1586663449829 7 connected
85a281c31f28583f9257d59c56353445d633ff72 127.0.0.1:7002@17002 master - 0 1586663450446 3 connected 10923-16383
19927db6574a0436be0e2b6850c26211f8800259 127.0.0.1:7004@17004 slave 85a281c31f28583f9257d59c56353445d633ff72 0 1586663450000 5 connected
f0ddc47c9674761642125fe6ee813e113c4b4717 127.0.0.1:7001@17001 master,fail - 1586663443522 1586663442597 2 disconnected

二、使用redis 编译目录下的工具

如果不想如上所述通过手动配置和执行各个实例来创建Redis集群,则可以使用一个简单得多的系统(但是您将不会学到相同数量的操作细节)。只需检查Redis发行版中的utils / create-cluster目录。 内部有一个名为create-cluster的脚本(名称与包含在其中的目录相同),它是一个简单的bash脚本。 为了启动具有3个主节点和3个从节点的6节点群集,只需键入以下命令:

create-cluster start
create-cluster create

当redis-cli实用程序希望您接受集群布局时,在步骤2中回复yes。现在,您可以与群集进行交互,默认情况下,第一个节点将从端口30001开始。 完成后,使用以下命令停止集群:

create-cluster stop

参考: 

    https://redis.io/topics/cluster-tutorial

 

  • 1
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值