Redis with HA

1. 概述

Redis(http://redis.io/)一个高性能的key-value数据库，可用于实时的计算和统计等。
Redis有多种HA方案，比如Master-Slave，Redis Cluster等方案。
本文采用Redis Sentinel来管理1 Master 2 Slave的3台Centos 6.3 VM实例的方式，每台VM实例有一个Redis进程和一个Sentinel进程。

对于小型的应用来说，此方案已经足够。

2. 部署配置

假设

redis master	10.64.70.10
redis slave 1	10.64.70.20
redis slave 2	10.64.70.30

创建以下目录结构

/data/redis/config	存放redis.conf sentinel.conf
/data/redis/backup/	存放定期持久化的backup db
/date/redis/log/	存放redis.log sentinel.log
/data/redis/logrotate/	存放redis logrotate config：redis.rotate

在3台VM上安装redis并拷贝配置文件

$ wget http://download.redis.io/releases/redis-3.0.5.tar.gz
$ tar xzf redis-3.0.5.tar.gz
$ cd redis-3.0.5
$ make
$ make install

拷贝redis-3.0.5.tar.gz包中原始的redis.conf sentinel.conf 到/data/redis/config/ 并修改3台VM上的配置文件

Redis master redis.conf

# Master redis.conf

daemonize yes
pidfile /var/run/redis.pid
port 6379
bind 10.64.70.10
timeout 0
loglevel notice
logfile /data/redis/log/redis.log
databases 16
#save 900 1
#save 300 10
#save 60 10000
rdbcompression yes
dbfilename redis_dump.db
dir /data/redis/backup
#slaveof
appendonly no
slave-serve-stale-data yes
slave-read-only yes

Redis salve1 redis.conf

# Slave1 redis.conf

daemonize yes
pidfile /var/run/redis.pid
port 6379
bind 10.64.70.20
timeout 0
loglevel notice
logfile /data/redis/log/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename redis_dump.db
dir /data/redis/backup
slaveof 10.64.70.10 6379
appendonly yes
slave-serve-stale-data yes
slave-read-only yes

Redis slave 2 redis.conf

# Slave2 redis.conf

daemonize yes
pidfile /var/run/redis.pid
port 6379
bind 10.64.70.30
timeout 0
loglevel notice
logfile /data/redis/log/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename redis_dump.db
dir /data/redis/backup
slaveof 10.64.70.10 6379
appendonly yes
slave-serve-stale-data yes
slave-read-only yes

Every sentinel.conf

# all Sentinel sentinel.conf

daemonize yes
loglevel notice
logfile /data/redis/log/sentinel.log
port 26379
sentinel monitor redis1 10.64.70.10 6379 2
sentinel monitor redis2 10.64.70.20 6379 2
sentinel monitor redis3 10.64.70.30 6379 2
sentinel down-after-milliseconds redis1 5000
sentinel down-after-milliseconds redis2 5000
sentinel down-after-milliseconds redis3 5000
sentinel parallel-syncs redis1 1
sentinel parallel-syncs redis2 1
sentinel parallel-syncs redis3 1
sentinel failover-timeout redis1 900000
sentinel failover-timeout redis2 900000
sentinel failover-timeout redis3 900000

Logrotate: /data/redis/logrotate/redis.rotate

/data/redis/log/redis.log
/data/redis/log/sentinel.log
{
daily
rotate 30
copytruncate
notifempty
missingok
dateext
dateformat -%Y%m%d
}

Create cron job “00 00 * * * /usr/sbin/logrotate /data/redis/logrotate/redis.rotate -f” to execute the rotate.

3. 启动关闭

启动，在3台上执行

redis-server /data/redis/config/redis.conf
redis-server /data/redis/config/sentinel.conf --sentinel

关闭redis，分别执行

redis-cli -h 10.64.70.10 shutdown
redis-cli -h 10.64.70.20 shutdown
redis-cli -h 10.64.70.30 shutdown

关闭sentinel，在3台上执行

redis-cli -p 26379 shutdown

4. 监控

简单的监控方式是判断每台上的redis和sentinel进程是否存在，不存在则发送alert出来。
判断存在的脚本的例子，可以由zenoss定期执行根据返回结果向设定好的收件人发alert。

#!/bin/sh

LOCAL_IP=$(/sbin/ifconfig eth0 | /bin/awk '/inet addr/{print substr($2,6)}')
REDIS_PROCESS_NUM=$(/usr/bin/pgrep -l -f redis-server | /usr/bin/wc -l)

if [ "$REDIS_PROCESS_NUM" == "2" ]; then
    echo "OK: [$LOCAL_IP]Redis running"
    exit 0
else
    echo "ERROR: [$LOCAL_IP]Redis stopped"
    exit 1        
fi

实时监控方案1: zenoss + Redis Monitor ZenPack
http://wiki.zenoss.org/ZenPack:Redis_Monitor

实时监控方案2: Nagios + plugin check_redis.pl
http://exchange.nagios.org/directory/Plugins/Databases/check_redis-2Epl/details
https://github.com/willixix/WL-NagiosPlugins