redis-sentinel安装、配置与故障与故障转移演示

redis-sentinel安装、配置与故障与故障转移演示

redis-sentinel安装、配置与故障与故障转移演示
本次搭建，将会搭建一个redis-sentinel，其中包括一主两从三哨兵。架构图如下：

+-------------+      +-------------+      +-------------+
| Sentinel S1 |------| Sentinel S2 |------| Sentinel S3 |
+-------------+      +-------------+      +-------------+
      |                    |                    |
      |                    |                    |
      +------+------+------+------+------+------+
                           |
                           |                     
                       +--------+
                       | Master |
                       +--------+
                           |
                           | 
            +------++------+------++------+
            |                             |
            |                             |           
      +-----------+                 +-----------+
      | Salve  S1 |                 | Salve  S2 |
      +-----------+                 +-----------+

1、环境

• Linux：CentOS release 6.5 (Final) Kernel
• redis：3.2.8

2、下载

通过https://redis.io/下载redis的稳定版本，本次是下载的3.2.8

3、解压与编译

将redis-3.2.8.tar.gz放入/data/app目录，并解压：

tar zxvf redis-3.2.8.tar.gz

进入/data/app/redis-3.2.8目录，编译：

make 

4、建立日志/pid/dump文件目录

进入/data/app/redis-3.2.8目录，建立如下目录

mkdir logs
mkdir pid
mkdir dump

5、修改配置文件

复制配置

cp redis.conf redis.conf_bak
cp redis.conf redis_6379.conf
cp redis.conf redis_6380.conf
cp redis.conf redis_6381.conf

cp sentinel.conf sentinel.conf_bak
cp sentinel.conf sentinel_26379.conf
cp sentinel.conf sentinel_26379.conf
cp sentinel.conf sentinel_26379.conf

其中

• 备份配置 ：redis.conf_bak

- 备份配置 ：sentinel.conf_bak

- master配置：redis_6379.conf

• slave1配置：redis_6380.conf

- slave2配置：redis_6381.conf

• sentinel1 : sentinel_26379.conf
• sentinel2 : sentinel_26380.conf
• sentinel3 : sentinel_26381.conf

5.1、master配置(redis_6379.conf)

# Redis configuration file example.
# 绑定IP地址
bind 192.168.10.242

# 设置为非保护模式
protected-mode no

# 设置端口
port 6379

# TCP listen() backlog.
tcp-backlog 511

# Unix socket.
timeout 0

# TCP keepalive.
tcp-keepalive 300

################################# GENERAL #####################################
# 设置为以守护进程方式
daemonize yes

supervised no

# 设置pid文件
pidfile /data/app/redis-3.2.8/pid/redis_6379.pid

loglevel notice

# 设置日志目录
logfile /data/app/redis-3.2.8/logs/redis_6379.log

databases 16

################################ SNAPSHOTTING  ################################
# 设置为不存储
# save 900 1
# save 300 10
# save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

# The filename where to dump the DB 
# 指定dump文件
dbfilename dump_6379.rdb

# The working directory. 
# 设置工作目录
dir /data/app/redis-3.2.8/dump/

################################# REPLICATION #################################
# slaveof <masterip> <masterport>
# masterauth <master-password>
# 设置master密码
masterauth admin

slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5

repl-disable-tcp-nodelay no

slave-priority 100

################################## SECURITY ###################################
# requirepass foobared
# 设置需要密码访问
requirepass admin

################################### LIMITS ####################################
# maxclients 10000
# maxmemory <bytes>

############################## APPEND ONLY MODE ###############################
appendonly no

appendfilename "appendonly.aof"


# appendfsync everysec
appendfsync no

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

aof-load-truncated yes

################################ LUA SCRIPTING  ###############################
lua-time-limit 5000

################################ REDIS CLUSTER  ###############################
slowlog-log-slower-than 10000

slowlog-max-len 128

################################ LATENCY MONITOR ##############################
latency-monitor-threshold 0

############################# EVENT NOTIFICATION ##############################
notify-keyspace-events ""

############################### ADVANCED CONFIG ###############################
hash-max-ziplist-entries 512
hash-max-ziplist-value 64

list-max-ziplist-size -2

list-compress-depth 0

set-max-intset-entries 512

zset-max-ziplist-entries 128
zset-max-ziplist-value 64

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

5.2、slave配置

(redis_6380.conf/redis_6381.conf)
将port,pidfile,logfile,dbfilename设置成与改端口对应的，另外需要加上如下配置：

slaveof 192.168.10.242 6379

5.3、sentinel配置

sentinel_26379.conf配置如下：

# Example sentinel.conf

# *** IMPORTANT ***
# port <sentinel-port>
# The port that this sentinel instance will run on
# 端口
port 26379

# sentinel announce-ip <ip>
# sentinel announce-port <port>

# 设置为非保护模式
protected-mode no

# sentinel announce-ip 1.2.3.4

# dir <working-directory
dir /tmp

# 设置为守护线程方式运行
daemonize yes

# 设置日志
logfile /data/app/redis-3.2.8/logs/sentinel_26379.log

# sentinel monitor <master-name> <ip> <redis-port> <quorum>
# 设置监听的master服务器(服务名，IP，端口，odown投票数)
sentinel monitor mymaster 192.168.10.242 6379 2

# sentinel auth-pass <master-name> <password>
# sentinel认证密码
sentinel auth-pass mymaster admin

# sentinel down-after-milliseconds <master-name> <milliseconds>
# Default is 30 seconds.
# sentinel down-after-milliseconds mymaster 30000
# 在一段时间内获取不到master的心跳时，将会认为该master为sdown状态，
# 此处的设置值就是这个一段时间值，设置为3秒，默认为30秒
sentinel down-after-milliseconds mymaster 3000

# sentinel parallel-syncs <master-name> <numslaves>
#
# How many slaves we can reconfigure to point to the new slave simultaneously
# during the failover. Use a low number if you use the slaves to serve query
# to avoid that all the slaves will be unreachable at about the same
# time while performing the synchronization with the master.
sentinel parallel-syncs mymaster 1

# sentinel failover-timeout <master-name> <milliseconds>
# Default is 3 minutes.
# sentinel failover-timeout mymaster 180000
# 故障转移时间，设置为5秒，默认为3分钟
sentinel failover-timeout mymaster 5000

# SCRIPTS EXECUTION
#
# NOTIFICATION SCRIPT
#
# sentinel notification-script <master-name> <script-path>
# 
# sentinel notification-script mymaster /var/redis/notify.sh
#
# CLIENTS RECONFIGURATION SCRIPT
#
# sentinel client-reconfig-script <master-name> <script-path>
#
# Example:
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

另外两个sentinel配置修改对应port，logfile配置即可。

6、启动redis-sentinel

进入/data/app/redis-3.2.8/src目录，通过下面命令启动redis-sentinel，先主后从，最后哨兵

./redis-server ../redis_6379.conf
./redis-server ../redis_6380.conf
./redis-server ../redis_6381.conf

./redis-sentinel ../sentinel_26379.conf
./redis-sentinel ../sentinel_26380.conf
./redis-sentinel ../sentinel_26381.conf

7、故障转移演示

前提：将redis-sentinel成功启动；

1. 如果master6379出现问题，sentinel1在设置的时间内无法获取到master6379的心跳，sentinel1投票master6379为sdown状态；将sentine投票数与quorum(配置文件中quorumw为2)比较，sentinel投票数小于quorum，未否达到了odown的条件，继续下一步；
2. sentinel2，在设置的时间内无法获取到master6379的心跳，sentinel2投票master6379为sdown状态，sentinel投票数增加，将sentinel投票数与quorum比较，sentinel投票数等于quorum，达到odown条件；
3. 开始进行故障转移
4. sentinel将会投票从salve6380和salve6381中选出新的master
5. 选举出master，更新sentinel配置文件
6. 故障转移完成
   注意：并不一定是sentinel1首先发现master6379有问题，本步骤只是说明这个过程；

进入/data/app/redis-3.2.8/src目录，将master6379 shutdown

./redis-cli -h 192.168.10.242 -p 6379 -a admin shutdown

故障转移过程：

27414:X 12 May 15:22:51.702 # Sentinel ID is bcd187b87f97488692628dff9ab94bdfddd11907
27414:X 12 May 15:22:51.703 # +monitor master mymaster 192.168.10.242 6379 quorum 2
27414:X 12 May 15:22:51.704 * +slave slave 192.168.10.242:6380 192.168.10.242 6380 @ mymaster 192.168.10.242 6379
27414:X 12 May 15:22:51.726 * +slave slave 192.168.10.242:6381 192.168.10.242 6381 @ mymaster 192.168.10.242 6379
27414:X 12 May 15:22:58.002 * +sentinel sentinel 3251d086105f7131e1c39267fae42f27770bc40f 192.168.10.242 26380 @ mymaster 192.168.10.242 6379
27414:X 12 May 15:23:01.595 * +sentinel sentinel b003b642473ebfca5453666b4d587216971e0d23 192.168.10.242 26381 @ mymaster 192.168.10.242 6379
// sentinel x 发现master6379状态异常，将master6379设置为sdown状态
27414:X 12 May 15:25:44.992 # +sdown master mymaster 192.168.10.242 6379 
// sentinel y 发现master6379状态异常，达到quorm条件，将master6379设置为odown状态
27414:X 12 May 15:25:45.047 # +odown master mymaster 192.168.10.242 6379 #quorum 3/2
27414:X 12 May 15:25:45.047 # +new-epoch 1
// 对master6379进行故障转移
27414:X 12 May 15:25:45.047 # +try-failover master mymaster 192.168.10.242 6379
// sentinel投票开始
27414:X 12 May 15:25:45.100 # +vote-for-leader bcd187b87f97488692628dff9ab94bdfddd11907 1
27414:X 12 May 15:25:45.100 # 3251d086105f7131e1c39267fae42f27770bc40f voted for 3251d086105f7131e1c39267fae42f27770bc40f 1
27414:X 12 May 15:25:45.200 # b003b642473ebfca5453666b4d587216971e0d23 voted for 3251d086105f7131e1c39267fae42f27770bc40f 1
// 3251d086105f7131e1c39267fae42f27770bc40f票数为2，成为新的master，更新sentinel配置文件
27414:X 12 May 15:25:46.245 # +config-update-from sentinel 3251d086105f7131e1c39267fae42f27770bc40f 192.168.10.242 26380 @ mymaster 192.168.10.242 6379
// 6381成为新的master，故障转移完成
27414:X 12 May 15:25:46.245 # +switch-master mymaster 192.168.10.242 6379 192.168.10.242 6381
27414:X 12 May 15:25:46.246 * +slave slave 192.168.10.242:6380 192.168.10.242 6380 @ mymaster 192.168.10.242 6381

8、其他命令参考

查看redis信息：

./redis-cli -h 192.168.10.242 -p 26379 -a admin info 

查看sentinel信息：

./redis-cli -h 192.168.10.242 -p 26379 -a admin info

将对应redis服务停止

./redis-cli -h 192.168.10.242 -p 6379 -a admin shutdown

9、遇到的问题

搭建完成后，无法连接上sentinel，telnet 192.168.10.242 26379错误信息如下：

-DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients.
In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the follow
ing solutions: 

1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, 
however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 
2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server.
3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 
4) Setup a bind address or an authentication password. 

NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.

解决方案：
在sentinel配置文件中加上：

# 设置为非保护模式
protected-mode no