keepalived配置redis主从、异常自动切换

前言

文章中记录的内容配置成功后可以达到异常后vip自动飘逸，并且redis可以自动切换主从关系，但还存在两个问题需要优化

1. keepalived-MASTER异常后VIP会飘逸到keepalived-BACKUP，但是只要keepalived-MASTER恢复正常，VIP一定会飘逸到keepalived-MASTER上，造成不必要的主备切换，对于这个情况网上有设置成双BACKUP，具体请百度
2. 采用这种方式存在数据丢失问题，只能通过优化notify_master、notify_backup脚本或者其他的方式尽可能地减少数据丢失

核心思想

使用keepalived监控redis，keepalived检测到redis异常后触发主备切换，执行keepalived的notify_master、notify_backup脚本进行相应处理。

notify_master：keepalived变为Master时调用

notify_backup：keepalived变为Slave时调用

配置/验证过程中实用的命令

• 实时查看keepalived日志

新开窗口执行上面命令，在另一个窗口中操作keepalived，可实时查看keepalived的日志，如果启动异常，日志会打印失败的信息

tail -f /var/log/messages

• 设置成功后查看虚拟ip

ip a

配置过程

1. 编写配置文件

keepalived官网配置文件解释：https://www.keepalived.org/manpage.html

• keepalived-MASTERr配置文件

! Configuration File for keepalived

global_defs {
	notification_email {
		xxx@qq.com
	}
	notification_email_from xxxx@163.com
	smtp_server mail.163.com
	smtp_connect_timeout 30
	router_id 195
	script_user root
	enable_script_security
}

vrrp_script redis_check {
	script "/opt/server/keepalived/scripts/redis_check.sh 127.0.0.1 6379"
	#检查时间间隔
	interval 1
	#脚本结果导致的优先级变更：10表示优先级+20；-20则表示优先级-20
	weight -20
	rise 3
	fall 3
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 1
    priority 100
	#心跳广播时间间隔(秒) 默认检测三次，延长3秒开始切换脚本,是为了让redis_backup/master.sh脚本有充足的时间执行完毕
    advert_int 1
	authentication {
		auth_type PASS
		auth_pass 1111
	}
	virtual_ipaddress {
		10.2.0.202
	}
	track_script{
		redis_check
	}
	#notify_master、notify_backup等脚本执行时如果要传入参数，他们的值必须用单引号包住
	notify_master '/opt/server/keepalived/scripts/redis_master.sh 10.2.0.196 6379'
	notify_backup '/opt/server/keepalived/scripts/redis_slave.sh 10.2.0.196 6379'
	notify_fault /opt/server/keepalived/scripts/redis_fault.sh
	notify_stop /opt/server/keepalived/scripts/redis_stop.sh
}

• keepalived-BACKUP配置文件

! Configuration File for keepalived

global_defs {
	notification_email {
		xxx@qq.com
	}
	notification_email_from xxx@163.com
	smtp_server mail.163.com
	smtp_connect_timeout 30
	router_id 195
	script_user root
	enable_script_security
}

vrrp_script redis_check {
	script "/opt/server/keepalived/scripts/redis_check.sh 127.0.0.1 6379"
	#检查时间间隔
	interval 1
	#脚本结果导致的优先级变更：10表示优先级+20；-20则表示优先级-20
	weight -20
	rise 3
	fall 3
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 1
    priority 90
	#心跳广播时间间隔(秒) 默认检测三次，延长3秒开始切换脚本,是为了让redis_backup/master.sh脚本有充足的时间执行完毕
    advert_int 1
    authentication {
		auth_type PASS
		auth_pass 1111
    }
	virtual_ipaddress {
		10.2.0.202
	}
	track_script{
		redis_check
	}
	#notify_master、notify_backup等脚本执行时如果要传入参数，他们的值必须用单引号包住
	notify_master '/opt/server/keepalived/scripts/redis_master.sh 10.2.0.195 6379'
	notify_backup '/opt/server/keepalived/scripts/redis_slave.sh 10.2.0.195 6379'
	notify_fault /opt/server/keepalived/scripts/redis_fault.sh
	notify_stop /opt/server/keepalived/scripts/redis_stop.sh
}

• redis_check.sh

#!/bin/bash
ALIVE=`/opt/server/redis-5.0.8/src/redis-cli -h $1 -p $2 PING`
LOGFILE="/opt/server/keepalived/logs/keepalived-redis-state.log"
echo "[CHECK]" >> $LOGFILE
date >> $LOGFILE
if [ $ALIVE == "PONG" ]; then :
   echo "Success: redis-cli -h $1 -p $2 PING $ALIVE" >> $LOGFILE 2>&1
    exit 0
else
    echo "Failed:redis-cli -h $1 -p $2 PING $ALIVE " >> $LOGFILE 2>&1
    exit 1
fi

• redis_master.sh

#!/bin/bash
REDISCLI="/opt/server/redis-5.0.8/src/redis-cli"
LOGFILE="/opt/server/keepalived/logs/"
LOGNAME="keepalived-redis-change-master.log"
if [ ! -d "/opt/server/keepalived/logs" ]; then
mkdir -p /opt/server/keepalived/logs
fi
echo "[master]" >> $LOGFILE$LOGNAME
date >> $LOGFILE$LOGNAME
echo "Run SLAVEOF Cmd , Start Sync Data ..." >> $LOGFILE$LOGNAME
$REDISCLI SLAVEOF $1 $2 >>$LOGFILE$LOGNAME  2>&1
#延迟10秒以后待数据同步完成后再取消同步状态
sleep 10
echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE$LOGNAME
$REDISCLI SLAVEOF NO ONE >> $LOGFILE$LOGNAME 2>&1

• redis_slave.sh

#!/bin/bash
REDISCLI="/opt/server/redis-5.0.8/src/redis-cli"
LOGFILE="/opt/server/keepalived/logs/"
LOGNAME="keepalived-redis-change-slave.log"
if [ ! -d "/opt/server/keepalived/logs" ]; then
mkdir -p /opt/server/keepalived/logs
fi
echo "[BACKUP]" >> $LOGFILE$LOGNAME
date >> $LOGFILE$LOGNAME
echo "Being slave wait 15 m for anthor sync data ...." >> $LOGFILE$LOGNAME
#延迟15秒待数据被对方同步完成之后再切换主从角色
sleep 15  
echo "Run SLAVEOF CMD ..." >> $LOGFILE$LOGNAME
$REDISCLI SLAVEOF $1 $2 >>$LOGFILE$LOGNAME  2>&1

• redis_fault.sh

#!/bin/bash
LOGFILE="/opt/server/keepalived/logs/keepalived-redis-state.log"
echo "[fault]" >> $LOGFILE
date >> $LOGFILE

• redis_stop.sh

#!/bin/bash
LOGFILE="/opt/server/keepalived/logs/keepalived-redis-state.log"
echo "[stop]" >> $LOGFILE
date >> $LOGFILE

2. 上传文件至相应目录并赋权

• redis_check.sh、redis_master.sh、redis_slave.sh、redis_fault.sh、redis_stop.sh文件赋予执行权限

# 上传在/opt/server/keepalived/scripts文件夹
chmod -R 0755 /opt/server/keepalived/scripts 

• redis_check.sh、redis_master.sh、redis_slave.sh、redis_fault.sh、redis_stop.sh文件赋予SEshell权限

chcon -t keepalived_unconfined_script_exec_t  /opt/server/keepalived/scripts/redis_master.sh
chcon -t keepalived_unconfined_script_exec_t  /opt/server/keepalived/scripts/redis_slave.sh
chcon -t keepalived_unconfined_script_exec_t  /opt/server/keepalived/scripts/redis_fault.sh
chcon -t keepalived_unconfined_script_exec_t  /opt/server/keepalived/scripts/redis_stop.sh

3. 启动主备redis

分别启动两个redis（不用配主动关系，脚本会自动设置）

/opt/server/redis-5.0.8/src/redis-server /opt/server/redis-5.0.8/redis-keepalived.conf

4. 启动keepalived

先启动keepalived-Master，在启动keepalived-Slave

systemctl start keepalived

5. 验证

1. 在主redis中设置内容，看是否同步到从redis
2. 停掉主redis，看vip是否漂移成功以及从redis是否成功设置为主
3. 恢复原来的主redis，看vip是否漂移成功

配置过程中碰到的问题

1. (/opt/server/keepalived/etc/keepalived/keepalived.conf: Line 41) (VI_1) track script not found, ignoring…

配置文件中track_script格式有问题，用Notrpad++打开配置文件，选中track_script的内容，移除前面所有空格（如下图），

再按tab进行缩进应有的格式（如下图），如果还有问题，将配置文件所有配置都按上面方式进行处理

2. Invalid user/group for notify script /opt/server/keepalived/scripts/redis_master.sh - ignoring

notify_master 、notify_slave等执行的脚本需要传递参数时要用单引号包住

例如：

notify_master ‘/opt/server/keepalived/scripts/redis_master.sh 10.2.0.195 6379’

3. 主备切换不执行notify_master、notify_backup、notify_fault、notify_stop

chcon -t keepalived_unconfined_script_exec_t /opt/server/keepalived/scripts/redis_master.sh

chcon -t keepalived_unconfined_script_exec_t /opt/server/keepalived/scripts/redis_slave.sh

chcon -t keepalived_unconfined_script_exec_t /opt/server/keepalived/scripts/redis_fault.sh

chcon -t keepalived_unconfined_script_exec_t /opt/server/keepalived/scripts/redis_stop.sh

3. Can‘t open PID file /run/keepalived.pid (yet?) after start: No such file or directory

原因是没有完全关闭服务，虽然systemctl显示停止了，但是还有残余的进程。执行以下命令

pkill keepalived