目前,项目用的redis主备,感觉超不爽,很多局限性,特别是在 redis master 宕机之后,维护非常麻烦,寻思着弄一个redis集群,Redis3.0+后终于提供了cluster方案。
架构:Redis Cluster采用无中心结构,每个节点都保存数据和整个集群的状态
每个节点都和其他所有节点连接,这些连接保持活跃
使用gossip协议传播信息以及发现新节点
node不作为client请求的代理,client根据node返回的错误信息重定向请求
数据分布:预分桶
预分好16384个桶,根据 CRC16(key) mod 16384的值,决定将一个key放到哪个桶中
每个Redis物理结点负责一部分桶的管理,当发生Redis节点的增减时,调整桶的分布即可
例如,假设Redis Cluster三个节点A/B/C,则
Node A 包含桶的编号可以为: 0 到 5500.
Node B 包含桶的编号可以为: 5500 到 11000.
Node C包含桶的编号可以为: 11001 到 16384.
当发生Redis节点的增减时,调整桶的分布即可。
预分桶的方案介于“硬Hash”和“一致性Hash”之间,牺牲了一定的灵活性,但相比“一致性Hash“,数据的管理成本大大降低
安装配置
安装编译环境
$yum install gcc gcc-c++ gcc-g77 flex bison autoconf automake bzip2-devel zlib-devel ncurses-devel libjpeg-devel libpng-devel libtiff-devel freetype-devel pam-devel openssl-devel libxml2-devel gettext-devel pcre-devel tcl -y
安装高版本ruby (yum安装的版本过低)
卸载老版本:
$yum erase ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs
$wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.1.tar.gz
$tar -zxvf ...
$./configure
$make && make install
$gem sources -r https://rubygems.org/
$gem sources -a https://ruby.taobao.org/
$gem sources –l
安装rubygems
$wget http://production.cf.rubygems.org/rubygems/rubygems-2.2.2.tgz
$ruby setup.rb
安装redis
$wget http://download.redis.io/releases/redis-3.2.0.tar.gz
$make MALLOC=libc
$cd src
$cp redis-server /usr/sbin/
$cp redis-cli /usr/sbin/
$cp redis-trib.rb /usr/sbin/
utils目录下有命令脚本的样例参考。
配置
- 创建用户和目录:
$adduser -M -U redis
$mkdir /var/lib/redis
$mkdir /etc/redis
$chown redis:redis /var/lib/redis
$chown redis:redis /etc/redis
- 修改系统配置:
$echo 1 > /proc/sys/vm/overcommit_memory
$vi /etc/sysctl.conf
添加vm.overcommit_memory=1
$echo 511 > /proc/sys/net/core/somaxconn
$echo never > /sys/kernel/mm/transparent_hugepage/enabled
$vi /etc/rc.local
添加echo never > /sys/kernel/mm/transparent_hugepage/enabled
编辑redis配置文件7000~7005
$vi /etc/redis/7000.conf
7000.conf(其它配置一致,port7000~7005)
$bind 127.0.0.1
$port 7000
$pidfile /var/run/redis/7000.pid
$loglevel notice
$logfile /var/log/redis/7000.log
$daemonize yes
$dir /var/lib/redis/7000
$cluster-enabled yes
$cluster-config-file nodes.conf
$cluster-node-timeout 5000
$appendonly yes
$appendfsync everysec
$repl-disable-tcp-nodelay yes
启动文件/etc/init.d/redis-cluster
>
#!/bin/bash
# Source function library.
. /etc/rc.d/init.d/functions
# Settings
PORT=7000
TIMEOUT=2000
NODES=6
REPLICAS=1
# You may want to put the above config parameters into config.sh in order to
# override the defaults without modifying this script.
if [ -a config.sh ]
then
source "config.sh"
fi
# Computed vars
ENDPORT=$((PORT+NODES))
if [ "$1" == "start" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
#PORT=$((PORT+1))
echo "Starting $PORT"
daemon --user redis "/usr/sbin/redis-server /etc/redis/${PORT}.conf"
PORT=$((PORT+1))
done
exit 0
fi
if [ "$1" == "create" ]
then
HOSTS=""
while [ $((PORT < ENDPORT)) != "0" ]; do
#PORT=$((PORT+1))
HOSTS="$HOSTS 127.0.0.1:$PORT"
PORT=$((PORT+1))
done
/usr/sbin/redis-trib.rb create --replicas $REPLICAS $HOSTS
exit 0
fi
if [ "$1" == "stop" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
#PORT=$((PORT+1))
echo "Stopping $PORT"
/usr/sbin/redis-cli -p $PORT shutdown nosave
PORT=$((PORT+1))
done
exit 0
fi
if [ "$1" == "watch" ]
then
while [ 1 ]; do
clear
date
PORTTMP=$PORT
while [ $((PORTTMP < ENDPORT)) != "0" ]; do
/usr/sbin/redis-cli -p $PORTTMP cluster nodes | head -30
PORTTMP=$((PORTTMP+1))
done
sleep 2
done
exit 0
fi
if [ "$1" == "tail" ]
then
INSTANCE=$2
PORT=$((PORT+INSTANCE))
tail -f /var/log/redis/${PORT}.log
exit 0
fi
if [ "$1" == "call" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
#PORT=$((PORT+1))
/usr/sbin/redis-cli -p $PORT $2 $3 $4 $5 $6 $7 $8 $9
PORT=$((PORT+1))
done
exit 0
fi
if [ "$1" == "clean" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
rm -rf /var/log/redis/${PORT}.log
rm -rf /var/lib/redis/${PORT}/*
PORT=$((PORT+1))
done
exit 0
fi
echo "Usage: $0 [start|create|stop|watch|tail|clean]"
echo "start -- Launch Redis Cluster instances."
echo "create -- Create a cluster using redis-trib create."
echo "stop -- Stop Redis Cluster instances."
echo "watch -- Show CLUSTER NODES output (first 30 lines) of first node."
echo "tail <id> -- Run tail -f of instance at base port + ID."
echo "clean -- Remove all instances data, logs, configs."
一些简单的命令
检查状态
$redis-trib.rb check 127.0.0.1:7000
添加master节点
$redis-trib.rb add-node 127.0.0.1:7006
node:新节点没有包含任何数据, 因为它没有包含任何slot。新加入的加点是一个主节点, 当集群需要将某个从节点升级为新的主节点时, 这个新节点不会被选中
为新节点分配slot
$redis-trib.rb reshard 127.0.0.1:7006
添加slave节点
步骤同上redis-cli连接上新节点shell,输入命令:cluster replicate 对应master的mode-id
>cluster replicate 2b9ebcbd627ff0fd7a7bbcc5332fb09e72788835
删除slave节点
$redis-trib.rb del-node ip:port '<node-id>' $redis-trib.rb del-node 127.0.0.1:7103 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx