Redis 3.0 Cluster配置文档
V2.0
说明:文档如有纰漏,欢迎提出指导建议,13482003931@163.com
文档控制
版本 | 内容 | 时间 | 作者 |
V1.0 | 初始版本 | 2015/4/23 | 张耐 |
V2.0 | 修复部分文字描述不准确问题 | | |
目录
Redis
架构拓扑
3.0版本最大的特点就是支持cluster分布式横向扩展,下面为3个Master节点以及3个slave节点的拓扑图:
APP1 |
准备阶段
(1)
http://redis.io
(2)
安装gcc:yum install gcc
安装zlib:yum install zib
安装ruby:yum install ruby
安装rubygems:yum install rubygems
安装gem redis:(下载:http://rubygems.org/gems/redis/versions/3.0.7)
# gem install -l /tmp/redis-3.0.7.gem
Successfully installed redis-3.0.7
1 gem installed
Installing ri documentation for redis-3.0.7...
Installing RDoc documentation for redis-3.0.7...
(3)
修改open files:# ulimit -n
添加vm.overcommit_memory=1:
#vi /etc/sysctl.conf
#sysctl vm.overcommit_memory=1
关闭hugepage:# echo never > /sys/kernel/mm/transparent_hugepage/enabled
修改somaxconn :# echo 511 >/proc/sys/net/core/somaxconn
关闭防火墙:# service iptables stop
关闭selinux:# vi /etc/sysconfig/selinux
安装Cluster
(1)
# cd /redis/redis-3.0.0
# make
# make install
(2)
# cp /redis/redis-3.0.0/src/redis-trib.rb /usr/local/bin/
# cp redis-cli /usr/local/bin/
# cp redis-server /usr/local/bin/
# which redis-trib.rb
/usr/local/bin/redis-trib.rb
(3)
# vi /redis/redis-3.0.0/config/redis-common.conf
代码如下: |
#GENERAL daemonize yes tcp-backlog 511 timeout 0 tcp-keepalive 0 loglevel notice databases 16 dir /redis/redis-3.0.0/data slave-serve-stale-data yes slave-read-only yes #not use default repl-disable-tcp-nodelay yes slave-priority 100 appendonly yes appendfsync everysec no-appendfsync-on-rewrite yes auto-aof-rewrite-min-size 64mb lua-time-limit 5000 cluster-enabled yes cluster-node-timeout 15000 cluster-migration-barrier 1 slowlog-log-slower-than 10000 slowlog-max-len 128 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-entries 512 list-max-ziplist-value 64 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 aof-rewrite-incremental-fsync yes |
(4)
# vi /redis/redis-3.0.0/config/redis-6379.conf
代码如下: |
include /redis/redis-3.0.0/config/redis-common.conf port 6379 logfile "/redis/redis-3.0.0/log/redis-6379.log" maxmemory 100m # volatile-lru -> remove the key with an expire set using an LRU algorithm # allkeys-lru -> remove any key accordingly to the LRU algorithm # volatile-random -> remove a random key with an expire set # allkeys-random -> remove a random key, any key # volatile-ttl -> remove the key with the nearest expire time (minor TTL) # noeviction -> don't expire at all, just return an error on write operations maxmemory-policy allkeys-lru appendfilename "appendonly-6379.aof" dbfilename dump-6379.rdb #dir /redis/redis-3.0.0/data cluster-config-file nodes-6379.conf auto-aof-rewrite-percentage 80-100 |
(5)
# vi /redis/redis-3.0.0/config/redis-6389.conf
代码如下: |
include /redis/redis-3.0.0/config/redis-common.conf port 6389 logfile "/redis/redis-3.0.0/log/redis-6389.log" maxmemory 100m # volatile-lru -> remove the key with an expire set using an LRU algorithm # allkeys-lru -> remove any key accordingly to the LRU algorithm # volatile-random -> remove a random key with an expire set # allkeys-random -> remove a random key, any key # volatile-ttl -> remove the key with the nearest expire time (minor TTL) # noeviction -> don't expire at all, just return an error on write operations maxmemory-policy allkeys-lru appendfilename "appendonly-6389.aof" dbfilename dump-6389.rdb cluster-config-file nodes-6389.conf auto-aof-rewrite-percentage 80-100 |
(6)
# vi /redis/redis-3.0.0/config/redis-6399.conf
代码如下: |
include /redis/redis-3.0.0/config/redis-common.conf port 6399 logfile "/redis/redis-3.0.0/log/redis-6399.log" maxmemory 100m # volatile-lru -> remove the key with an expire set using an LRU algorithm # allkeys-lru -> remove any key accordingly to the LRU algorithm # volatile-random -> remove a random key with an expire set # allkeys-random -> remove a random key, any key # volatile-ttl -> remove the key with the nearest expire time (minor TTL) # noeviction -> don't expire at all, just return an error on write operations maxmemory-policy allkeys-lru appendfilename "appendonly-6399.aof" dbfilename dump-6399.rdb cluster-config-file nodes-6399.conf auto-aof-rewrite-percentage 80-100 |
(7)
# redis-server redis-6379.conf
# redis-server redis-6389.conf
# redis-server redis-6399.conf
# redis-server redis-7379.conf
# redis-server redis-7389.conf
# redis-server redis-7399.conf
# ps -ef| grep redis
root
root
root
root
root
root
(8)
#--replicas
# redis-trib.rb create --replicas 1 192.168.3.88:6379 192.168.3.88:6389 192.168.3.88:6399 192.168.3.88:7379 192.168.3.88:7389 192.168.3.88:7399
>>> Creating cluster
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:7379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7399: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.3.88:6379
192.168.3.88:6389
192.168.3.88:6399
Adding replica 192.168.3.88:7379 to 192.168.3.88:6379
Adding replica 192.168.3.88:7389 to 192.168.3.88:6389
Adding replica 192.168.3.88:7399 to 192.168.3.88:6399
M: 05fe758161e2cbe23240697f
M: d1d124d35c848e9c8e726b59
M: d64223d6695fcc7e1030f219
S: 7f77ec03e40d0cc9f343d783
S: 98dae5126228dea54d1321ee
S: d013aee7cae8163f787cb644
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join......
>>> Performing Cluster Check (using node 192.168.3.88:6379)
M: 05fe758161e2cbe23240697f
M: d1d124d35c848e9c8e726b59
M: d64223d6695fcc7e1030f219
M: 7f77ec03e40d0cc9f343d783
M: 98dae5126228dea54d1321ee
M: d013aee7cae8163f787cb644
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
(9)
连接任意节点,执行redis-trib.rb
# redis-trib.rb check 192.168.3.88:6379
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7399: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:7379: OK
>>> Performing Cluster Check (using node 192.168.3.88:6379)
M: 05fe758161e2cbe23240697f
S: 98dae5126228dea54d1321ee
S: d013aee7cae8163f787cb644
M: d1d124d35c848e9c8e726b59
M: d64223d6695fcc7e1030f219
S: 7f77ec03e40d0cc9f343d783
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
管理cluster
添加的master节点配置在另一个服务器上,首先配置config文件
# vi /redis/redis-3.0.0/config/redis-6379.conf
# vi /redis/redis-3.0.0/config/redis-7379.conf
使用redis-trib.rb添加节点
在已有集群服务器(192.168.3.88)上执行
(注意:add-node的使用方法为new_host:new_port existing_host:existing_port,前面是新添加的节点信息,后面是已存在的节点信息)
# redis-trib.rb add-node 192.168.3.61:6379 192.168.3.88:6379
>>> Adding node 192.168.3.61:6379 to cluster 192.168.3.88:6379
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7399: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:7379: OK
>>> Performing Cluster Check (using node 192.168.3.88:6379)
M: 05fe758161e2cbe23240697f
S: 98dae5126228dea54d1321ee
S: d013aee7cae8163f787cb644
M: d1d124d35c848e9c8e726b59
M: d64223d6695fcc7e1030f219
S: 7f77ec03e40d0cc9f343d783
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Connecting to node 192.168.3.61:6379: OK
>>> Send CLUSTER MEET to node 192.168.3.61:6379 to make it join the cluster.
[OK] New node added correctly.
选择其中一个节点,检查集群状态,发现刚添加的节点已经在集群中了,角色是master,但是并没有slot分配到新加的节点上,后面要通过shard命令分配slot。
# redis-trib.rb check 192.168.3.88:6379
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7399: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.61:6379: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:7379: OK
>>> Performing Cluster Check (using node 192.168.3.88:6379)
M: 05fe758161e2cbe23240697f
S: 98dae5126228dea54d1321ee
S: d013aee7cae8163f787cb644
M: d1d124d35c848e9c8e726b59
M: 89be535ff56586dcec56f141
M: d64223d6695fcc7e1030f219
S: 7f77ec03e40d0cc9f343d783
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
主节点添加完毕后,需要给该主节点添加一个slave节点,添加过程和主节点一直,添加完成后需要在redis中进行设置。
# redis-trib.rb add-node 192.168.3.61:7379 192.168.3.88:6379
显示内容省略
链接到要添加的slave数据库中,执行replicate操作。后面的ID为Master 192.168.3.61:6379的ID,通过redis-trib.rb check可以看到。
# redis-cli -c -h 192.168.3.61 -p 7379
192.168.3.61:7379> cluster replicate 89be535ff56586dcec56f141
OK
根据check结果,可以看到新添加的slave以及成功和Master建立联系。
# redis-trib.rb check 192.168.3.88:6379
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7399: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.61:6379: OK
Connecting to node 192.168.3.61:7379: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:7379: OK
>>> Performing Cluster Check (using node 192.168.3.88:6379)
M: 05fe758161e2cbe23240697f
S: 98dae5126228dea54d1321ee
S: d013aee7cae8163f787cb644
M: d1d124d35c848e9c8e726b59
M: 89be535ff56586dcec56f141
S: 92017f0258675b02a7799726
M: d64223d6695fcc7e1030f219
S: 7f77ec03e40d0cc9f343d783
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
加入新的节点后,需要将其他的节点中的hash slot移动到新的节点中,以达到负载均衡的效果,指定集群中其中一个节点的地址
# redis-trib.rb reshard 192.168.3.6379
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7399: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.61:6379: OK
Connecting to node 192.168.3.61:7379: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:7379: OK
>>> Performing Cluster Check (using node 192.168.3.88:6379)
M: 05fe758161e2cbe23240697f
S: 98dae5126228dea54d1321ee
S: d013aee7cae8163f787cb644
M: d1d124d35c848e9c8e726b59
M: 89be535ff56586dcec56f141
S: 92017f0258675b02a7799726
M: d64223d6695fcc7e1030f219
S: 7f77ec03e40d0cc9f343d783
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?4096
What is the receiving node ID? 89be535ff56586dcec56f141
Please enter all the source node IDs.
Source node #1:all
Moving slot 12284 from 192.168.3.88:6399 to 192.168.3.61:6379:
Moving slot 12285 from 192.168.3.88:6399 to 192.168.3.61:6379:
Moving slot 12286 from 192.168.3.88:6399 to 192.168.3.61:6379:
Moving slot 12287 from 192.168.3.88:6399 to 192.168.3.61:6379:
再次check,发现所有主节点的slot都变成4096了
# redis-trib.rb check 192.168.3.88:6379
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7399: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.61:6379: OK
Connecting to node 192.168.3.61:7379: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:7379: OK
>>> Performing Cluster Check (using node 192.168.3.88:6379)
M: 05fe758161e2cbe23240697f
S: 98dae5126228dea54d1321ee
S: d013aee7cae8163f787cb644
M: d1d124d35c848e9c8e726b59
M: 89be535ff56586dcec56f141
S: 92017f0258675b02a7799726
M: d64223d6695fcc7e1030f219
S: 7f77ec03e40d0cc9f343d783
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
删除主节点之前,需要先将slot迁移到其他主节点上
# redis-trib.rb reshard 192.168.3.88:6379
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7399: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.61:6379: OK
Connecting to node 192.168.3.61:7379: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:7379: OK
>>> Performing Cluster Check (using node 192.168.3.88:6379)
M: 05fe758161e2cbe23240697f
S: 98dae5126228dea54d1321ee
S: d013aee7cae8163f787cb644
M: d1d124d35c848e9c8e726b59
M: 89be535ff56586dcec56f141
S: 92017f0258675b02a7799726
M: d64223d6695fcc7e1030f219
S: 7f77ec03e40d0cc9f343d783
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4906
What is the receiving node ID? 89be535ff56586dcec56f141
Please enter all the source node IDs.
Source node #1:d1d124d35c848e9c8e726b59
Source node #2:done
……
……
Do you want to proceed with the proposed reshard plan (yes/no)?yes
……
……
Moving slot 10920 from 192.168.3.88:6389 to 192.168.3.61:6379:
Moving slot 10921 from 192.168.3.88:6389 to 192.168.3.61:6379:
Moving slot 10922 from 192.168.3.88:6389 to 192.168.3.61:6379:
检查节点的slot是否完全迁移走,完成后就可以删除节点了
# redis-trib.rb check 192.168.3.88:6399
# redis-trib.rb del-node 192.168.3.88:6399 d64223d6695fcc7e1030f219
>>> Removing node d64223d6695fcc7e1030f219
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.61:6379: OK
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7379: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.61:7379: OK
Connecting to node 192.168.3.88:7399: OK
>>> Sending CLUSTER FORGET messages to the cluster...
>>> 192.168.3.88:7399 as replica of 192.168.3.88:6399
/usr/lib/ruby/gems/1.8/gems/redis-3.0.7/lib/redis/client.rb:97:in `call': ERR Can't forget my master!(Redis::CommandError)
删除主节点之前,需要先删除主节点的slave节点,否则会报如上错误
# redis-trib.rb del-node 192.168.3.88:7399 d013aee7cae8163f787cb644
>>> Removing node d013aee7cae8163f787cb644
Connecting to node 192.168.3.88:7399: OK
Connecting to node 192.168.3.61:6379: OK
Connecting to node 192.168.3.61:7379: OK
Connecting to node 192.168.3.88:7379: OK
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:6389: OK
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
# redis-trib.rb del-node 192.168.3.88:6399 d64223d6695fcc7e1030f219
>>> Removing node d64223d6695fcc7e1030f219
Connecting to node 192.168.3.88:6399: OK
Connecting to node 192.168.3.61:6379: OK
Connecting to node 192.168.3.88:6379: OK
Connecting to node 192.168.3.88:7389: OK
Connecting to node 192.168.3.88:7379: OK
Connecting to node 192.168.3.88:6389: OK
Connecting to node 192.168.3.61:7379: OK
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.