redis(一)：搭建redis高可用集群

一. 搭建环境

操作系统：CentOS7
内存：2GB
核心数：单核
redis版本：5.0.7
集群结构：3主3从

二. redis安装

# 解压redis
tar -xzvf redis-5.0.7.tar.gz
cd redis-5.0.7
# 编译并且安装
make && make install

# 启动redis，查看是否安装成功
/usr/local/bin/redis-server /usr/local/src/redis-5.0.7/redis.conf
# 客户端连接
/usr/local/bin/redis-cli -h 127.0.0.1

三. 集群搭建

我们这里只使用了一台机器搭建了一个伪集群，条件允许的话可以采用6台服务器搭建
在/opt目录下，当然也可以自己定义目录，创建6个文件夹，用来启动6个redis应用

cd /opt
mkdir myredis
cd myredis
mkdir redis7000
mkdir redis7001
mkdir redis7002
mkdir redis7003
mkdir redis7004
mkdir redis7005

拷贝一份redis.conf到redis7000目录下

cp /usr/local/src/redis-5.0.7/redis.conf /opt/myredis/redis7000

编辑redis.conf配置文件

[root@localhost redis-5.0.7]# cd /opt/myredis/redis7000/
[root@localhost redis7000]# vim redis.conf

编辑配置文件内容如下：

# 绑定IP
bind 192.168.252.20
# 绑定端口
port 7000
# 保护模式，是否允许外网访问，当redis设置了密码连接时，设置为yes外网也可以连接
protected-mode yes
# 守护进程模式开启
daemonize yes
# 为当前redis进程指定一个运行pid，因为我们是在同一台服务器上运行6个，所以需要区分以下
pidfile /var/run/redis_7000.pid
# 日志文件
logfile "/opt/myredis/redis7000/redis.log"
# 数据库个数，默认是16
databases 16
# rdb持久化保存策略，第二个参数为时间间隔，第三个参数为发生变化的key的个数，及60秒内有10000个key发生了变化，redis将会进行一次持久化，将数据保存到硬盘上，保存的数据可以在redis启动时加载回内存
save 900 1
save 300 10
save 60 10000
# 在持久化时发生错误是否停止写入数据，默认为yes
stop-writes-on-bgsave-error yes
# 持久化数据的文件名 
dbfilename dump.rdb
# 工作目录，持久化的数据将会保存到该目录下，如果采用默认的./，那么redis在哪个文件下启动，数据就会写到哪个文件夹，这样会发生数据错乱，因此最好指定为绝对路径
dir /opt/myredis/redis7000/
# 主节点授权密码，每个redis必须设置为相同的密码，这样当redis主节点挂掉后选择从节点时才能校验通过
masterauth 123456
# redis自身的密码
requirepass 123456
# 是否开启aof持久化机制，aof开启时，优先加载aof持久化文件，不会加载rdb的文件
# aof与rdb的区别：
# 1.aof在宕机时损失的数据更少，aof每秒会持久化对redis的操作
# 2.aof记录的是完整的操作过程，而不是像rdb持久化所有的数据，因此性能开销更小
# 3.aof记录过程不会启动子进程，而rdb会启动一个子进程来持久化数据，如果频率太高，会造成对服务的性能开销
appendonly no
# aof持久化文件名
appendfilename "appendonly.aof"
# aof持久化策略，默认为everysec
# everysec: 每秒，有可能会丢失1秒的数据
# no: 没有策略，让操作系统自己决定是否持久化，性能高，但不可控
# always: 每写入一次都进行一次aof，数据不丢失，但性能差
# 一般采用折衷方案everysec
appendfsync everysec
# 重写时不进行aof
no-appendfsync-on-rewrite no
# 重写比例，即当64mb触发了重写后，下次会在这个基础上64 + 32mb时才会触发重写
auto-aof-rewrite-percentage 100
# 触发重写的最小大小，即当内存达到64mb的时候触发重写，重写会整合冗余的操作，比如set k1 v1, del k1, 这种会被合并
auto-aof-rewrite-min-size 64mb
# 开启集群
cluster-enabled yes
# 集群配置文件，后续集群组建成功后会保存所有的节点信息到该文件
cluster-config-file nodes-7000.conf
# 如果一个master挂掉了，并且没有node可以升级为master，那么默认的该小集群不可用，其他的节点可用，如果设置为yes，那么所有节点都不可用
cluster-require-full-coverage no
# 子进程根据内存快照，按照命令合并规则写入到新的AOF文件中。每次批量写入磁盘的数据量由aof-rewrite-incremental-fsync参数控制，默认为32M，避免单次刷盘数据过多造成硬盘阻塞。
aof-rewrite-incremental-fsync yes

配置文件修改好后，替换端口号并写入到其他5个文件夹

cd /opt/myredis/redis7000
sed 's/7000/7001/g' redis.conf > ../redis7001/redis.conf

至此配置文件结束

四. 启动集群

首先，启动6个redis实例

/usr/local/bin/redis-server /opt/myredis/redis7000/redis.conf

利用redis-cli创建集群
首先，查看cluster集群帮助

[root@localhost redis7000]# /usr/local/bin/redis-cli -h 192.168.252.20 -p 7000 -a 123456 cluster help
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
 1) CLUSTER <subcommand> arg arg ... arg. Subcommands are:
 2) ADDSLOTS <slot> [slot ...] -- Assign slots to current node.
 3) BUMPEPOCH -- Advance the cluster config epoch.
 4) COUNT-failure-reports <node-id> -- Return number of failure reports for <node-id>.
 5) COUNTKEYSINSLOT <slot> - Return the number of keys in <slot>.
 6) DELSLOTS <slot> [slot ...] -- Delete slots information from current node.
 7) FAILOVER [force|takeover] -- Promote current replica node to being a master.
 8) FORGET <node-id> -- Remove a node from the cluster.
 9) GETKEYSINSLOT <slot> <count> -- Return key names stored by current node in a slot.
10) FLUSHSLOTS -- Delete current node own slots information.
11) INFO - Return onformation about the cluster.
12) KEYSLOT <key> -- Return the hash slot for <key>.
13) MEET <ip> <port> [bus-port] -- Connect nodes into a working cluster.
14) MYID -- Return the node id.
15) NODES -- Return cluster configuration seen by node. Output format:
16)     <id> <ip:port> <flags> <master> <pings> <pongs> <epoch> <link> <slot> ... <slot>
17) REPLICATE <node-id> -- Configure current node as replica to <node-id>.
18) RESET [hard|soft] -- Reset current node (default: soft).
19) SET-config-epoch <epoch> - Set config epoch of current node.
20) SETSLOT <slot> (importing|migrating|stable|node <node-id>) -- Set slot state.
21) REPLICAS <node-id> -- Return <node-id> replicas.
22) SLOTS -- Return information about slots range mappings. Each range is made of:
23)     start, end, master and replicas IP addresses, ports and ids

查看集群操作帮助

[root@localhost redis7000]# /usr/local/bin/redis-cli -a 123456 --cluster help
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Cluster Manager Commands:
  create         host1:port1 ... hostN:portN
                 --cluster-replicas <arg>
  check          host:port
                 --cluster-search-multiple-owners
  info           host:port
  fix            host:port
                 --cluster-search-multiple-owners
  reshard        host:port
                 --cluster-from <arg>
                 --cluster-to <arg>
                 --cluster-slots <arg>
                 --cluster-yes
                 --cluster-timeout <arg>
                 --cluster-pipeline <arg>
                 --cluster-replace
  rebalance      host:port
                 --cluster-weight <node1=w1...nodeN=wN>
                 --cluster-use-empty-masters
                 --cluster-timeout <arg>
                 --cluster-simulate
                 --cluster-pipeline <arg>
                 --cluster-threshold <arg>
                 --cluster-replace
  add-node       new_host:new_port existing_host:existing_port
                 --cluster-slave
                 --cluster-master-id <arg>
  del-node       host:port node_id
  call           host:port command arg arg .. arg
  set-timeout    host:port milliseconds
  import         host:port
                 --cluster-from <arg>
                 --cluster-copy
                 --cluster-replace
  help           

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
 

利用create命令操作

/usr/local/bin/redis-cli -a 123456 --cluster create 192.168.252.20:7000 192.168.252.20:7001 192.168.252.20:7002 192.168.252.20:7003 192.168.252.20:7004 192.168.252.20:7005 --cluster-replicas 1

查看节点信息,通过节点信息可以看出，节点角色，从节点对应的的主节点，主节点分配的槽位

[root@localhost redis7000]# /usr/local/bin/redis-cli -a 123456 -h 192.168.252.20 -p 7000 cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
72155559ed525e082af75bad9b40f3136b9b2f41 192.168.252.20:7002@17002 master - 0 1578557267500 3 connected 10923-16383
80266034d9f8131d3b1435588e7b9655948a64c0 192.168.252.20:7005@17005 slave c8e74726e0250c9107b73843ec1c33445bc86414 0 1578557264475 6 connected
282567d3c12b0dc72883f88048efed25504b211b 192.168.252.20:7001@17001 master - 0 1578557263467 2 connected 5461-10922
c8e74726e0250c9107b73843ec1c33445bc86414 192.168.252.20:7000@17000 myself,master - 0 1578557264000 1 connected 0-5460
7420db30b9dce45c9b065e3c5c1a5749f560f02a 192.168.252.20:7004@17004 slave 72155559ed525e082af75bad9b40f3136b9b2f41 0 1578557265000 5 connected
715b3f5c67b2940bc065a46a1e1c724c7af3d3df 192.168.252.20:7003@17003 slave 282567d3c12b0dc72883f88048efed25504b211b 0 1578557266492 4 connected

五. 集群扩容与缩容

5.1 扩容

相同的方式创建redis7006,redis7007文件夹即配置文件，并且启动redis

/usr/local/bin/redis-server /opt/myredis/redis7006/redis.conf
/usr/local/bin/redis-server /opt/myredis/redis7007/redis.conf

添加节点到集群中，此时新加入的master没有分配槽位

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 -h 192.168.252.20 -p 7000 cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
72155559ed525e082af75bad9b40f3136b9b2f41 192.168.252.20:7002@17002 master - 0 1578559296000 10 connected 10923-16383
80266034d9f8131d3b1435588e7b9655948a64c0 192.168.252.20:7005@17005 slave c8e74726e0250c9107b73843ec1c33445bc86414 0 1578559297000 8 connected
282567d3c12b0dc72883f88048efed25504b211b 192.168.252.20:7001@17001 master - 0 1578559298095 11 connected 5461-10922
9bbb545f3d3d90962d2dce79314c90ec044c105e 192.168.252.20:7007@17007 master - 0 1578559297088 0 connected
c8e74726e0250c9107b73843ec1c33445bc86414 192.168.252.20:7000@17000 myself,master - 0 1578559294000 8 connected 0-5460
7420db30b9dce45c9b065e3c5c1a5749f560f02a 192.168.252.20:7004@17004 slave 72155559ed525e082af75bad9b40f3136b9b2f41 0 1578559296000 10 connected
715b3f5c67b2940bc065a46a1e1c724c7af3d3df 192.168.252.20:7003@17003 slave 282567d3c12b0dc72883f88048efed25504b211b 0 1578559295000 11 connected

为新增加的master分配槽位

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 --cluster reshard 192.168.252.20 7000
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 192.168.252.20:7000)
M: c8e74726e0250c9107b73843ec1c33445bc86414 192.168.252.20:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 72155559ed525e082af75bad9b40f3136b9b2f41 192.168.252.20:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 80266034d9f8131d3b1435588e7b9655948a64c0 192.168.252.20:7005
   slots: (0 slots) slave
   replicates c8e74726e0250c9107b73843ec1c33445bc86414
M: 282567d3c12b0dc72883f88048efed25504b211b 192.168.252.20:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 9bbb545f3d3d90962d2dce79314c90ec044c105e 192.168.252.20:7007
   slots: (0 slots) master
S: 7420db30b9dce45c9b065e3c5c1a5749f560f02a 192.168.252.20:7004
   slots: (0 slots) slave
   replicates 72155559ed525e082af75bad9b40f3136b9b2f41
S: 715b3f5c67b2940bc065a46a1e1c724c7af3d3df 192.168.252.20:7003
   slots: (0 slots) slave
   replicates 282567d3c12b0dc72883f88048efed25504b211b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 

输入需要分配的槽位数：16384/4 = 4096

What is the receiving node ID? 9bbb545f3d3d90962d2dce79314c90ec044c105e

输入接收槽位的节点ID：9bbb545f3d3d90962d2dce79314c90ec044c105e
输入需要转移槽位的节点ID，输入all表示全部都参与分配

Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: all

分配完成后再次查看节点信息，槽位信息变成了0-1364 5461-6826 10923-12287

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 -h 192.168.252.20 -p 7000 cluster nodes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
72155559ed525e082af75bad9b40f3136b9b2f41 192.168.252.20:7002@17002 master - 0 1578559795503 10 connected 12288-16383
80266034d9f8131d3b1435588e7b9655948a64c0 192.168.252.20:7005@17005 slave c8e74726e0250c9107b73843ec1c33445bc86414 0 1578559794000 8 connected
282567d3c12b0dc72883f88048efed25504b211b 192.168.252.20:7001@17001 master - 0 1578559792000 11 connected 6827-10922
9bbb545f3d3d90962d2dce79314c90ec044c105e 192.168.252.20:7007@17007 master - 0 1578559794496 12 connected 0-1364 5461-6826 10923-12287
c8e74726e0250c9107b73843ec1c33445bc86414 192.168.252.20:7000@17000 myself,master - 0 1578559795000 8 connected 1365-5460
7420db30b9dce45c9b065e3c5c1a5749f560f02a 192.168.252.20:7004@17004 slave 72155559ed525e082af75bad9b40f3136b9b2f41 0 1578559796509 10 connected
715b3f5c67b2940bc065a46a1e1c724c7af3d3df 192.168.252.20:7003@17003 slave 282567d3c12b0dc72883f88048efed25504b211b 0 1578559793000 11 connected

添加从节点，指定该节点为从节点，指定对应的master节点ID

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 --cluster add-node 192.168.252.20:7006 192.168.252.20:7000 --cluster-slave --cluster-master-id 9bbb545f3d3d90962d2dce79314c90ec044c105e

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 192.168.252.20:7006 to cluster 192.168.252.20:7000
>>> Performing Cluster Check (using node 192.168.252.20:7000)
M: c8e74726e0250c9107b73843ec1c33445bc86414 192.168.252.20:7000
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
M: 72155559ed525e082af75bad9b40f3136b9b2f41 192.168.252.20:7002
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: 80266034d9f8131d3b1435588e7b9655948a64c0 192.168.252.20:7005
   slots: (0 slots) slave
   replicates c8e74726e0250c9107b73843ec1c33445bc86414
M: 282567d3c12b0dc72883f88048efed25504b211b 192.168.252.20:7001
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
M: 9bbb545f3d3d90962d2dce79314c90ec044c105e 192.168.252.20:7007
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
S: 7420db30b9dce45c9b065e3c5c1a5749f560f02a 192.168.252.20:7004
   slots: (0 slots) slave
   replicates 72155559ed525e082af75bad9b40f3136b9b2f41
S: 715b3f5c67b2940bc065a46a1e1c724c7af3d3df 192.168.252.20:7003
   slots: (0 slots) slave
   replicates 282567d3c12b0dc72883f88048efed25504b211b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.252.20:7006 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 192.168.252.20:7007.
[OK] New node added correctly.

节点扩容成功！

5.2 缩容

为了验证缩容是数据跟随槽位移动，我们添加key为 k2的数据，数据经过crc16后得到449，数据自动重定向到192.168.252.20:7007上

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 -h 192.168.252.20 -p 7000 -c
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.252.20:7000> set k2 v2
-> Redirected to slot [449] located at 192.168.252.20:7007
OK
192.168.252.20:7007> exit

首先先从集群中删除从节点，这里最好先删除从节点，以免容灾机制启动，从节点自动升级为主节点，升级过程中小集群是不能正常使用的

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 --cluster del-node 192.168.252.20:7000 ab0672d8fae04355009d1f1a27285c720d5937da
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node ab0672d8fae04355009d1f1a27285c720d5937da from cluster 192.168.252.20:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

将需要缩容的节点槽位移动到其他节点

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 --cluster reshard 192.168.252.20:7000 --cluster-from 9bbb545f3d3d90962d2dce79314c90ec044c105e --cluster-to c8e74726e0250c9107b73843ec1c33445bc86414 --cluster-slots 1365

迁移完成后，槽位数为0

9bbb545f3d3d90962d2dce79314c90ec044c105e 192.168.252.20:7007@17007 master - 0 1578560624972 12 connected

此时，我们再次去get一次k2，确认k2已经随槽位迁移

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 -h 192.168.252.20 -p 7000 -c
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.252.20:7000> get k2
"v2"

上面可以看到，k2依然能够查询到，说明数据已经迁移，这时我们可以放心删除原来节点，
同时，可以删除节点的配置信息以及持久化数据，如果不删除，下次再要将这两个节点加入集群会报错，提示有数据，或者节点已经在其他集群

[root@localhost redis7007]# /usr/local/bin/redis-cli -a 123456 --cluster del-node 192.168.252.20:7000 9bbb545f3d3d90962d2dce79314c90ec044c105e
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node 9bbb545f3d3d90962d2dce79314c90ec044c105e from cluster 192.168.252.20:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

至此，缩容成功！！