Redis 3.2集群环境搭建和测试

Redis 3开始支持了Cluster模式，增强了Redis的水平扩展能力，Redis Cluster的节点分片通过hash slot实现，每个节点上的键都属于16384(0~16383)个slots中的一个，每个节点负责处理一部分slots。 Redis Cluster采用无中心节点方式实现，无需proxy代理，客户端直接与redis集群的每个节点连接，根据同样的hash算法计算出key对应的slot，然后直接在slot对应的Redis上执行命令。

Redis安装

---

环境准备

• CentOS 7.2
  • redis1 192.168.61.51 6379(master) 6380(slave)
  • redis2 192.168.61.52 6379(master) 6380(slave)
  • redis3 192.168.61.53 6379(master) 6380(slave)

• 修改内核参数vm.overcommit_memory = 1

vi /etc/sysctl.conf
vm.overcommit_memory = 1
  
sysctl -p

 编译安装Redis

wget http://download.redis.io/releases/redis-3.2.12.tar.gz
tar -zxvf redis-3.2.12.tar.gz
cd redis-3.2.12
make
make install PREFIX=/usr/local/redis

mkdir /usr/local/redis/etc
cp redis.conf /usr/local/redis/etc/

测试启动：

/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis.conf

集群初始化

配置

redis_6379.conf

daemonize yes
pidfile /var/run/redis_6379.pid
dir /var/lib/redis/redis_6379  
logfile /var/log/redis/redis_6379.log
appendonly yes
bind 192.168.61.51
# bind 192.168.61.52
# bind 192.168.61.53
port 6379

cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-node-timeout 15000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

redis_6380.conf

daemonize yes
pidfile /var/run/redis_6380.pid
dir /var/lib/redis/redis_6380  
logfile /var/log/redis/redis_6380.log
appendonly yes
bind 192.168.61.51
# bind 192.168.61.52
# bind 192.168.61.53
port 6380

cluster-enabled yes
cluster-config-file nodes-6380.conf
cluster-node-timeout 15000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

集群相关的配置：

• cluster-enabled 是否开启集群配置
• cluster-config-file 集群节点集群信息配置文件,每个节点都有一个,由redis生成和更新,配置时避免名称冲突
• cluster-node-timeout 集群节点互连超时的阀值，单位毫秒
• cluster-slave-validity-factor 进行故障转移时,salve会 申请成为master。有时slave会和master失联很久导致数据较旧，这样的slave不应该成为master。这个配置用来判断slave是否和master失联时间过长。

groupadd redis
useradd -g redis -d /var/lib/redis -s /sbin/nologin redis
mkdir -p /var/log/redis
mkdir -p /var/lib/redis/redis_6379
mkdir -p /var/lib/redis/redis_6380
chown -R redis:redis /var/log/redis /var/lib/redis

redis_6379.service

[Unit]
Description=redis
After=network.target

[Service]
Type=forking
User=redis
ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_6379.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

redis_6380.service

[Unit]
Description=redis
After=network.target

[Service]
Type=forking
User=redis
ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_6380.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

启动节点

systemctl start redis_6379 redis_6380

主节点

配置成功后，在各个主机上查看redis进程，各个redis节点都运行在cluster模式下：

ps -ef | grep redis
redis     2656     1  0 15:09 ?        00:00:00 /usr/local/redis/bin/redis-server 192.168.61.51:6379 [cluster]
redis     2666     1  0 15:09 ?        00:00:00 /usr/local/redis/bin/redis-server 192.168.61.51:6380 [cluster]

使用cluster meet添加主节点:

./redis-cli -h 192.168.61.51 -p 6379 -c

cluster meet 192.168.61.52 6379
cluster meet 192.168.61.53 6379

查看集群状态：

cluster info

cluster_state:fail  #集群状态
cluster_slots_assigned:0 #分配的slot数
cluster_slots_ok:0       #正确的slot数
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3    #当前的节点数
cluster_size:0
cluster_current_epoch:2
cluster_my_epoch:0
cluster_stats_messages_sent:171
cluster_stats_messages_received:171

上面显示cluster的状态是fail，是由于没有分配slot，需要把16384分配到3个Node上，编写addslots脚本：

addslots.sh

#!/bin/bash
for ((i=$2;i<=$3;i++))
do
   /usr/local/redis/bin/redis-cli -h $1 -p 6379 CLUSTER ADDSLOTS $i
done

分别在redis1,redis2,redis3三个主机上执行:

./addslots.sh 192.168.61.51 0 5461
./addslots.sh 192.168.61.52 5462 10922
./addslots.sh 192.168.61.53 10923 16383

再次查看集群状态：

CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:2
cluster_my_epoch:0
cluster_stats_messages_sent:3495
cluster_stats_messages_received:3495

从节点

先添加从节点到集群中：

cluster meet 192.168.61.51 6380
cluster meet 192.168.61.52 6380
cluster meet 192.168.61.53 6380

CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6  #所有节点数6个
cluster_size:3  #主节点数3个
cluster_current_epoch:5
cluster_my_epoch:0
cluster_stats_messages_sent:4388
cluster_stats_messages_received:4388

查看各节点id:

CLUSTER NODES
5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 192.168.61.53:6380 master - 0 1488959931731 5 connected
8f60e94a47da98286795b2fbbcae1b3aea75e123 192.168.61.51:6380 master - 0 1488959933751 3 connected
bd503b887831e642ac850af3fdd6e27c27086db7 192.168.61.51:6379 myself,master - 0 0 0 connected 0-5461
956f9f95b96bb02842609cea5dcda0cf89bfb78e 192.168.61.53:6379 master - 0 1488959935772 2 connected 10923-16383
877a1fad6a6c344a69707a335d2dcf94afeecbbd 192.168.61.52:6380 master - 0 1488959934761 4 connected
7290136f19a26c877c31f565068ea02badf11339 192.168.61.52:6379 master - 0 1488959936786 1 connected 5462-10922

下面分配从节点，主从关系如下：

master 192.168.61.51 6379  -> slave 192.168.61.52 6380
master 192.168.61.52 6379  -> slave 192.168.61.53 6380
master 192.168.61.53 6379  -> slave 192.168.61.51 6380

./redis-cli -h 192.168.61.51 -p 6380 -c
CLUSTER REPLICATE 956f9f95b96bb02842609cea5dcda0cf89bfb78e ##此处原文章有误
 

./redis-cli -h 192.168.61.52 -p 6380 -c
CLUSTER REPLICATE bd503b887831e642ac850af3fdd6e27c27086db7

./redis-cli -h 192.168.61.53 -p 6380 -c
CLUSTER REPLICATE 7290136f19a26c877c31f565068ea02badf11339

 CLUSTER NODES
877a1fad6a6c344a69707a335d2dcf94afeecbbd 192.168.61.52:6380 slave bd503b887831e642ac850af3fdd6e27c27086db7 0 1488960443865 4 connected
7290136f19a26c877c31f565068ea02badf11339 192.168.61.52:6379 master - 0 1488960447913 1 connected 5462-10922
956f9f95b96bb02842609cea5dcda0cf89bfb78e 192.168.61.53:6379 master - 0 1488960444893 2 connected 10923-16383
5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 192.168.61.53:6380 myself,slave 7290136f19a26c877c31f565068ea02badf11339 0 0 5 connected
8f60e94a47da98286795b2fbbcae1b3aea75e123 192.168.61.51:6380 slave 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 0 1488960446907 5 connected
bd503b887831e642ac850af3fdd6e27c27086db7 192.168.61.51:6379 master - 0 1488960445904 0 connected 0-5461

到这一步Redis 3的高可用和分配集群搭建完毕。

 

基于ruby的集群管理工具

---

安装ruby

安装ruby：

参照：centos7/rhel7安装较高版本ruby2.2/2.3/2.4+

安装rubygems:

wget https://rubygems.org/rubygems/rubygems-2.7.8.tgz
tar -zxvf rubygems-2.7.8.tgz
cd rubygems-2.7.8
ruby setup.rb

gem -v
2.7.8

 安装zlib：

wget http://www.zlib.net/zlib-1.2.11.tar.gz
tar -zxvf zlib-1.2.11.tar.gz
cd zlib-1.2.11
./configure --prefix=/usr/local/zlib
make
make install

安装 ruby-zlib:

cd ruby-2.4.0/ext/zlib
ruby ./extconf.rb --with-zlib-dir=/usr/local/zlib
make
make install

上面的命令在make时报make: *** No rule to make target /include/ruby.h', needed by zlib.o'. Stop.。修改Makefile文件：

ORIG_SRCS = /root/source/ruby-2.4.0/ext/zlib/zlib.c
SRCS = $(ORIG_SRCS)
OBJS = /root/source/ruby-2.4.0/ext/zlib/zlib.o

 安装redis gem：

gem sources --add http://rubygems.org
gem sources --remove https://rubygems.org/
gem sources list

gem q -rn redis
gem install redis

redis源码下的src/redis-trib.rb即为集群管理工具：

./redis-trib.rb
Usage: redis-trib <command> <options> <arguments ...>

  create          host1:port1 ... hostN:portN
                  --replicas <arg>
  check           host:port
  info            host:port
  fix             host:port
                  --timeout <arg>
  reshard         host:port
                  --from <arg>
                  --to <arg>
                  --slots <arg>
                  --yes
                  --timeout <arg>
                  --pipeline <arg>
  rebalance       host:port
                  --weight <arg>
                  --auto-weights
                  --use-empty-masters
                  --timeout <arg>
                  --simulate
                  --pipeline <arg>
                  --threshold <arg>
  add-node        new_host:new_port existing_host:existing_port
                  --slave
                  --master-id <arg>
  del-node        host:port node_id
  set-timeout     host:port milliseconds
  call            host:port command arg arg .. arg
  import          host:port
                  --from <arg>
                  --copy
                  --replace
  help            (show this help)

redis-trib.rb用法参考：redis cluster管理工具redis-trib.rb详解

文章转自：https://blog.frognew.com/2017/03/redis-3.2-cluster-install-and-test.html#%E9%9B%86%E7%BE%A4%E5%88%9D%E5%A7%8B%E5%8C%96