1. 环境准备
1.1. 硬件配置
标准化项 | 说明 | 备注 |
---|---|---|
IO设置 | 写策略设置Write Back,禁用Write Through | |
IO子系统设置 | 使用Raid1+0 | 若采用PCIe-SSD,则可不用做Raid |
关闭物理磁盘cache,防止丢数据 | ||
写策略设置Write Back,禁用Write Through | ||
关闭预读,使得宝贵的cahce用来作为写缓存 | ||
BIOS标准设置 | 开启最大性能模式,选择DAPC或者选择Performance模式 | 不同的主板配置上有差异 |
Memory Frequency(内存频率)选择Maximum Performance模式 | 不同的主板配置上有差异 | |
关闭NUMA | 不同的主板配置上有差异 | |
文件系统 | 数据存储文件系统类型建议EXT4格式 | 该项不做强制要求XFS类型也可以 |
网络配置 | 使用mode=4模式的网卡绑定 | 双网卡连接不同的交换机,交换机上需要做适配 |
数据挂载目录 | /data | Redis数据库实例的所有文件全部存放在/data目录下; |
1.2. 操作系统软件包&第三方工具
软件包 | 说明 | 备注 |
---|---|---|
gcc | 至少是C89兼容的 | |
GNU make | 版本3.80及以上 | |
tar | 解压源码包时用到的tar解压工具 |
1.3. 关闭SELINUX
临时禁用selinux:
# setenforce 0
# more /etc/selinux/config | grep -Ev "$|[#;]"
永久禁用selinux:
# vi /etc/selinux/config
将
SELINUX= enforcing
修改为
SELINUX=disabled
:x保存退出
1.4. 配置防火墙
如需要禁用防火墙,可执行如下命令:
1)若是CentOS 7.x系统,可执行如下命令禁用防火墙:
关闭防火墙:
# systemctl stop firewalld.service
禁用防火墙开机自启:
# systemctl disable firewalld.service
查看防火墙状态:
# systemctl status firewalld.service
2)若是Ubuntu系统,可执行如下命令禁用防火墙:
# iptables -P INPUT ACCEPT
3)其他系统禁用防火墙命令,请自行查找。
如需配置防火墙规则,可参照如下示例:
# vim /etc/iptables.rules
*filter
:INPUT ACCEPT [3715:523951]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1925:209933]
-A INPUT -i lo -j ACCEPT
#允许ssh端口通过
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
#允许icmp
-A INPUT -p icmp -j ACCEPT
#允许redis服务器间互通
-A INPUT -s 192.168.11.11/24 -p tcp -m state --state NEW -m tcp --dport 6801:6840 -j ACCEPT
-A INPUT -s 192.168.11.12/24 -p tcp -m state --state NEW -m tcp --dport 6801:6840 -j ACCEPT
-A INPUT -s 192.168.11.13/24 -p tcp -m state --state NEW -m tcp --dport 6801:6840 -j ACCEPT
-A INPUT -s 192.168.11.14/24 -p tcp -m state --state NEW -m tcp --dport 6801:6840 -j ACCEPT
#运行其他服务器访问Redis,如允许172.172.122.222服务器访问本地Redis端口为6801:6840的Redis实例
-A INPUT -s 172.172.122.222/32 -p tcp -m state --state NEW -m tcp --dport 6801:6840 -j ACCEPT
#暴露本地22201服务端口
-A INPUT -p tcp -m tcp --dport 22201 -j ACCEPT
-A INPUT -m state --state ESTABLISHED -j ACCEPT
COMMIT
防火墙规则配置完毕后,将配置文件导入:
# iptables-restore < /etc/iptables.rules
导入防火墙规则配置文件后,打开防火墙:
# iptables -P INPUT DROP
如需要开机自动恢复规则,把恢复命令添加到启动脚本:
# echo '/sbin/iptables-restore < /etc/iptables.rules' >>/etc/rc.local
# echo '/sbin/iptables -P INPUT DROP' >>/etc/rc.local
# cat /etc/rc.local
1.5. 优化系统参数
编辑系统配置文件sysctl.conf:
# vim /etc/sysctl.conf
vm.overcommit_memory=1
vm.swappiness = 0
net.core.somaxconn = 1024
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_fin_timeout = 30
net.nf_conntrack_max = 655350
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.ip_local_reserved_ports = 16001,16801,6801-6840,20001-20016
^: net.ipv4.tcp_tw_reuse设置为1表示开启重用,允许将TIME-WAIT sockets重新用于新的TCP连接,默认为0,表示关闭。
装载ip_conntrack模块:
# modprobe ip_conntrack
载入sysctl配置文件:
# sysctl -p
1.6. 修改hugepage配置
临时禁用hugepage:
# echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled
为了使机器重启后THP配置依然生效,可以在/etc/rc.local中追加:
# echo '/bin/echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
1.7. 修改环境变量
vim /etc/profile
......
unset i
unset -f pathmunge
export LD_LIBRARY_PATH=./
ulimit -SHn 655350
source /etc/profile
2. 部署redis
2.1. 节点规划
主节点:
192.168.11.11:6801-6820
192.168.11.12:6801-6820
192.168.11.13:6801-6820
192.168.11.14:6801-6820
从节点:
192.168.11.13:6821-6840
192.168.11.14:6821-6840
192.168.11.11:6821-6840
192.168.11.12:6821-6840
2.2. 安装依赖包
# apt-get install make
# apt-get install gcc
2.3. 安装Redis
2.3.1. 解压安装包、创建安装目录
解压安装包:
# cd /opt/app/softwares/
# tar zxvf redis-4.0.14.tar.gz -C /var/lib
编译安装:
# cd /var/lib/redis-4.0.14
# make MALLOC=libc
创建Redis部署所需目录:
# mkdir -p /data/redis/16801/dump
# mkdir -p /data/redis/16801/etc
# mkdir -p /data/redis/16801/log
# mkdir -p /data/redis/16801/run
2.3.2. 编辑配置文件
用脚本批量生成 6801-6840 端口的配置文件(需提前准备一个示例配置文件):
# vim /data/redis/16801/etc/cp_conf.sh
#!/bin/bash
i=6801
for ((;$i <= 6840; i=$i+1))
{
cp 6000.conf $i.conf
sed -i "s,6001,$i,g" $i.conf
sleep 1
}
为脚本赋予执行权限:
# chmod +x /data/redis/16801/etc/cp_conf.sh
^: 从 ubuntu 6.10 开始,ubuntu 就将先前默认的bash shell 更换成了dash shell;其表现为 /bin/sh 链接倒了/bin/dash而不是传统的/bin/bash。需执行dpkg-reconfigure dash,选择NO,即将默认的shell改回bash。
执行脚本,批量生成 6801-6840 端口的配置文件:
# cd /data/redis/16801/etc/
# sh /data/redis/16801/etc/cp_conf.sh
删除示例配置文件:
# rm -rf 6000.conf
2.3.3. 启动redis
配置环境变量:
# vim /etc/profile
export PATH=/var/lib/redis-4.0.14/src:$PATH
使配置生效:
# source /etc/profile
批量启动Redis:
# cd /var/lib/redis-4.0.14
# ./start-redis.sh -n 16801
查看进程:
# ps -ef| grep redis-server|grep -v grep
3. 配置Redis主从
按照 “2.1. 节点规划”的规划配置Redis主从复制:
# redis-cli -h 192.168.11.11 -p 6821 slaveof 192.168.11.12 6801
# redis-cli -h 192.168.11.11 -p 6822 slaveof 192.168.11.12 6802
# redis-cli -h 192.168.11.11 -p 6823 slaveof 192.168.11.12 6803
# redis-cli -h 192.168.11.11 -p 6824 slaveof 192.168.11.12 6804
# redis-cli -h 192.168.11.11 -p 6825 slaveof 192.168.11.12 6805
# redis-cli -h 192.168.11.11 -p 6826 slaveof 192.168.11.12 6806
# redis-cli -h 192.168.11.11 -p 6827 slaveof 192.168.11.12 6807
# redis-cli -h 192.168.11.11 -p 6828 slaveof 192.168.11.12 6808
# redis-cli -h 192.168.11.11 -p 6829 slaveof 192.168.11.12 6809
# redis-cli -h 192.168.11.11 -p 6830 slaveof 192.168.11.12 6810
# redis-cli -h 192.168.11.11 -p 6831 slaveof 192.168.11.12 6811
# redis-cli -h 192.168.11.11 -p 6832 slaveof 192.168.11.12 6812
# redis-cli -h 192.168.11.11 -p 6833 slaveof 192.168.11.12 6813
# redis-cli -h 192.168.11.11 -p 6834 slaveof 192.168.11.12 6814
# redis-cli -h 192.168.11.11 -p 6835 slaveof 192.168.11.12 6815
# redis-cli -h 192.168.11.11 -p 6836 slaveof 192.168.11.12 6816
# redis-cli -h 192.168.11.11 -p 6837 slaveof 192.168.11.12 6817
# redis-cli -h 192.168.11.11 -p 6838 slaveof 192.168.11.12 6818
# redis-cli -h 192.168.11.11 -p 6839 slaveof 192.168.11.12 6819
# redis-cli -h 192.168.11.11 -p 6840 slaveof 192.168.11.12 6820
# redis-cli -h 192.168.11.12 -p 6821 slaveof 192.168.11.13 6801
# redis-cli -h 192.168.11.12 -p 6822 slaveof 192.168.11.13 6802
# redis-cli -h 192.168.11.12 -p 6823 slaveof 192.168.11.13 6803
# redis-cli -h 192.168.11.12 -p 6824 slaveof 192.168.11.13 6804
# redis-cli -h 192.168.11.12 -p 6825 slaveof 192.168.11.13 6805
# redis-cli -h 192.168.11.12 -p 6826 slaveof 192.168.11.13 6806
# redis-cli -h 192.168.11.12 -p 6827 slaveof 192.168.11.13 6807
# redis-cli -h 192.168.11.12 -p 6828 slaveof 192.168.11.13 6808
# redis-cli -h 192.168.11.12 -p 6829 slaveof 192.168.11.13 6809
# redis-cli -h 192.168.11.12 -p 6830 slaveof 192.168.11.13 6810
# redis-cli -h 192.168.11.12 -p 6831 slaveof 192.168.11.13 6811
# redis-cli -h 192.168.11.12 -p 6832 slaveof 192.168.11.13 6812
# redis-cli -h 192.168.11.12 -p 6833 slaveof 192.168.11.13 6813
# redis-cli -h 192.168.11.12 -p 6834 slaveof 192.168.11.13 6814
# redis-cli -h 192.168.11.12 -p 6835 slaveof 192.168.11.13 6815
# redis-cli -h 192.168.11.12 -p 6836 slaveof 192.168.11.13 6816
# redis-cli -h 192.168.11.12 -p 6837 slaveof 192.168.11.13 6817
# redis-cli -h 192.168.11.12 -p 6838 slaveof 192.168.11.13 6818
# redis-cli -h 192.168.11.12 -p 6839 slaveof 192.168.11.13 6819
# redis-cli -h 192.168.11.12 -p 6840 slaveof 192.168.11.13 6820
# redis-cli -h 192.168.11.13 -p 6821 slaveof 192.168.11.14 6801
# redis-cli -h 192.168.11.13 -p 6822 slaveof 192.168.11.14 6802
# redis-cli -h 192.168.11.13 -p 6823 slaveof 192.168.11.14 6803
# redis-cli -h 192.168.11.13 -p 6824 slaveof 192.168.11.14 6804
# redis-cli -h 192.168.11.13 -p 6825 slaveof 192.168.11.14 6805
# redis-cli -h 192.168.11.13 -p 6826 slaveof 192.168.11.14 6806
# redis-cli -h 192.168.11.13 -p 6827 slaveof 192.168.11.14 6807
# redis-cli -h 192.168.11.13 -p 6828 slaveof 192.168.11.14 6808
# redis-cli -h 192.168.11.13 -p 6829 slaveof 192.168.11.14 6809
# redis-cli -h 192.168.11.13 -p 6830 slaveof 192.168.11.14 6810
# redis-cli -h 192.168.11.13 -p 6831 slaveof 192.168.11.14 6811
# redis-cli -h 192.168.11.13 -p 6832 slaveof 192.168.11.14 6812
# redis-cli -h 192.168.11.13 -p 6833 slaveof 192.168.11.14 6813
# redis-cli -h 192.168.11.13 -p 6834 slaveof 192.168.11.14 6814
# redis-cli -h 192.168.11.13 -p 6835 slaveof 192.168.11.14 6815
# redis-cli -h 192.168.11.13 -p 6836 slaveof 192.168.11.14 6816
# redis-cli -h 192.168.11.13 -p 6837 slaveof 192.168.11.14 6817
# redis-cli -h 192.168.11.13 -p 6838 slaveof 192.168.11.14 6818
# redis-cli -h 192.168.11.13 -p 6839 slaveof 192.168.11.14 6819
# redis-cli -h 192.168.11.13 -p 6840 slaveof 192.168.11.14 6820
# redis-cli -h 192.168.11.14 -p 6821 slaveof 192.168.11.11 6801
# redis-cli -h 192.168.11.14 -p 6822 slaveof 192.168.11.11 6802
# redis-cli -h 192.168.11.14 -p 6823 slaveof 192.168.11.11 6803
# redis-cli -h 192.168.11.14 -p 6824 slaveof 192.168.11.11 6804
# redis-cli -h 192.168.11.14 -p 6825 slaveof 192.168.11.11 6805
# redis-cli -h 192.168.11.14 -p 6826 slaveof 192.168.11.11 6806
# redis-cli -h 192.168.11.14 -p 6827 slaveof 192.168.11.11 6807
# redis-cli -h 192.168.11.14 -p 6828 slaveof 192.168.11.11 6808
# redis-cli -h 192.168.11.14 -p 6829 slaveof 192.168.11.11 6809
# redis-cli -h 192.168.11.14 -p 6830 slaveof 192.168.11.11 6810
# redis-cli -h 192.168.11.14 -p 6831 slaveof 192.168.11.11 6811
# redis-cli -h 192.168.11.14 -p 6832 slaveof 192.168.11.11 6812
# redis-cli -h 192.168.11.14 -p 6833 slaveof 192.168.11.11 6813
# redis-cli -h 192.168.11.14 -p 6834 slaveof 192.168.11.11 6814
# redis-cli -h 192.168.11.14 -p 6835 slaveof 192.168.11.11 6815
# redis-cli -h 192.168.11.14 -p 6836 slaveof 192.168.11.11 6816
# redis-cli -h 192.168.11.14 -p 6837 slaveof 192.168.11.11 6817
# redis-cli -h 192.168.11.14 -p 6838 slaveof 192.168.11.11 6818
# redis-cli -h 192.168.11.14 -p 6839 slaveof 192.168.11.11 6819
# redis-cli -h 192.168.11.14 -p 6840 slaveof 192.168.11.11 6820
主从配置完成后,查看各个节点的角色:
# for((i=6801;i<=6820;i++));do echo "==========192.168.11.11 : ${i}=========="; # redis-cli -h 192.168.11.11 -p ${i} role; done
# for((i=6801;i<=6820;i++));do echo "==========192.168.11.12 : ${i}=========="; # redis-cli -h 192.168.11.12 -p ${i} role; done
# for((i=6801;i<=6820;i++));do echo "==========192.168.11.13 : ${i}=========="; # redis-cli -h 192.168.11.13 -p ${i} role; done
# for((i=6801;i<=6820;i++));do echo "==========192.168.11.14 : ${i}=========="; # redis-cli -h 192.168.11.14 -p ${i} role; done
4. 部署twemproxy
4.1. 配置twemproxy
创建安装目录:
# mkdir -p /test/nutcracker/
编辑配置文件:
# vim /test/nutcracker/nutcracker/bin/conf/nutcracker.yml
alpha:
listen: 0.0.0.0:16801
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
preconnect: true
server_connections: 30
auto_eject_hosts: false
server_failure_limit: 2
server_retry_timeout: 30000
timeout: 500
redis: true
servers:
- 192.168.11.11:6801:1 server1
- 192.168.11.11:6802:1 server2
- 192.168.11.11:6803:1 server3
- 192.168.11.11:6804:1 server4
- 192.168.11.11:6805:1 server5
- 192.168.11.11:6806:1 server6
- 192.168.11.11:6807:1 server7
- 192.168.11.11:6808:1 server8
- 192.168.11.11:6809:1 server9
- 192.168.11.11:6810:1 server10
- 192.168.11.11:6811:1 server11
- 192.168.11.11:6812:1 server12
- 192.168.11.11:6813:1 server13
- 192.168.11.11:6814:1 server14
- 192.168.11.11:6815:1 server15
- 192.168.11.11:6816:1 server16
- 192.168.11.11:6817:1 server17
- 192.168.11.11:6818:1 server18
- 192.168.11.11:6819:1 server19
- 192.168.11.11:6820:1 server20
- 192.168.11.12:6801:1 server21
- 192.168.11.12:6802:1 server22
- 192.168.11.12:6803:1 server23
- 192.168.11.12:6804:1 server24
- 192.168.11.12:6805:1 server25
- 192.168.11.12:6806:1 server26
- 192.168.11.12:6807:1 server27
- 192.168.11.12:6808:1 server28
- 192.168.11.12:6809:1 server29
- 192.168.11.12:6810:1 server30
- 192.168.11.12:6811:1 server31
- 192.168.11.12:6812:1 server32
- 192.168.11.12:6813:1 server33
- 192.168.11.12:6814:1 server34
- 192.168.11.12:6815:1 server35
- 192.168.11.12:6816:1 server36
- 192.168.11.12:6817:1 server37
- 192.168.11.12:6818:1 server38
- 192.168.11.12:6819:1 server39
- 192.168.11.12:6820:1 server40
- 192.168.11.13:6801:1 server41
- 192.168.11.13:6802:1 server42
- 192.168.11.13:6803:1 server43
- 192.168.11.13:6804:1 server44
- 192.168.11.13:6805:1 server45
- 192.168.11.13:6806:1 server46
- 192.168.11.13:6807:1 server47
- 192.168.11.13:6808:1 server48
- 192.168.11.13:6809:1 server49
- 192.168.11.13:6810:1 server50
- 192.168.11.13:6811:1 server51
- 192.168.11.13:6812:1 server52
- 192.168.11.13:6813:1 server53
- 192.168.11.13:6814:1 server54
- 192.168.11.13:6815:1 server55
- 192.168.11.13:6816:1 server56
- 192.168.11.13:6817:1 server57
- 192.168.11.13:6818:1 server58
- 192.168.11.13:6819:1 server59
- 192.168.11.13:6820:1 server60
- 192.168.11.14:6801:1 server68
- 192.168.11.14:6802:1 server62
- 192.168.11.14:6803:1 server63
- 192.168.11.14:6804:1 server64
- 192.168.11.14:6805:1 server65
- 192.168.11.14:6806:1 server66
- 192.168.11.14:6807:1 server67
- 192.168.11.14:6808:1 server68
- 192.168.11.14:6809:1 server69
- 192.168.11.14:6810:1 server70
- 192.168.11.14:6811:1 server71
- 192.168.11.14:6812:1 server72
- 192.168.11.14:6813:1 server73
- 192.168.11.14:6814:1 server74
- 192.168.11.14:6815:1 server75
- 192.168.11.14:6816:1 server76
- 192.168.11.14:6817:1 server77
- 192.168.11.14:6818:1 server78
- 192.168.11.14:6819:1 server79
- 192.168.11.14:6820:1 server80
4.2. 启动twemproxy
创建日志目录:
# mkdir -p /log/
启动twemproxy:
# cd /test/nutcracker/nutcracker/bin/
# sh start.sh
查看进程:
# ps -ef| grep nutcracker|grep -v grep
# ps -ef| grep nutcracker|grep -v grep|wc -l
5. 部署哨兵
创建哨兵日志目录:
# mkdir /etc/sentinel
5.1. 编辑哨兵文件
# vim /etc/sentinel/sentinel.conf
port 26880
daemonize yes
dir "/tmp"
pidfile "/var/run/sentinel.pid"
loglevel notice
logfile "/var/log/sentinel.log"
protected-mode no
sentinel monitor test_16801_g1 192.168.11.11 6801 2
sentinel down-after-milliseconds test_16801_g1 100000
sentinel failover-timeout test_16801_g1 60000
sentinel client-reconfig-script test_16801_g1 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g2 192.168.11.11 6802 2
sentinel down-after-milliseconds test_16801_g2 100000
sentinel failover-timeout test_16801_g2 60000
sentinel client-reconfig-script test_16801_g2 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g3 192.168.11.11 6803 2
sentinel down-after-milliseconds test_16801_g3 100000
sentinel failover-timeout test_16801_g3 60000
sentinel client-reconfig-script test_16801_g3 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g4 192.168.11.11 6804 2
sentinel down-after-milliseconds test_16801_g4 100000
sentinel failover-timeout test_16801_g4 60000
sentinel client-reconfig-script test_16801_g4 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g5 192.168.11.11 6805 2
sentinel down-after-milliseconds test_16801_g5 100000
sentinel failover-timeout test_16801_g5 60000
sentinel client-reconfig-script test_16801_g5 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g6 192.168.11.11 6806 2
sentinel down-after-milliseconds test_16801_g6 100000
sentinel failover-timeout test_16801_g6 60000
sentinel client-reconfig-script test_16801_g6 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g7 192.168.11.11 6807 2
sentinel down-after-milliseconds test_16801_g7 100000
sentinel failover-timeout test_16801_g7 60000
sentinel client-reconfig-script test_16801_g7 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g8 192.168.11.11 6808 2
sentinel down-after-milliseconds test_16801_g8 100000
sentinel failover-timeout test_16801_g8 60000
sentinel client-reconfig-script test_16801_g8 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g9 192.168.11.11 6809 2
sentinel down-after-milliseconds test_16801_g9 100000
sentinel failover-timeout test_16801_g9 60000
sentinel client-reconfig-script test_16801_g9 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g10 192.168.11.11 6810 2
sentinel down-after-milliseconds test_16801_g10 100000
sentinel failover-timeout test_16801_g10 60000
sentinel client-reconfig-script test_16801_g10 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g11 192.168.11.11 6811 2
sentinel down-after-milliseconds test_16801_g11 100000
sentinel failover-timeout test_16801_g11 60000
sentinel client-reconfig-script test_16801_g11 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g12 192.168.11.11 6812 2
sentinel down-after-milliseconds test_16801_g12 100000
sentinel failover-timeout test_16801_g12 60000
sentinel client-reconfig-script test_16801_g12 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g13 192.168.11.11 6813 2
sentinel down-after-milliseconds test_16801_g13 100000
sentinel failover-timeout test_16801_g13 60000
sentinel client-reconfig-script test_16801_g13 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g14 192.168.11.11 6814 2
sentinel down-after-milliseconds test_16801_g14 100000
sentinel failover-timeout test_16801_g14 60000
sentinel client-reconfig-script test_16801_g14 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g15 192.168.11.11 6815 2
sentinel down-after-milliseconds test_16801_g15 100000
sentinel failover-timeout test_16801_g15 60000
sentinel client-reconfig-script test_16801_g15 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g16 192.168.11.11 6816 2
sentinel down-after-milliseconds test_16801_g16 100000
sentinel failover-timeout test_16801_g16 60000
sentinel client-reconfig-script test_16801_g16 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g17 192.168.11.11 6817 2
sentinel down-after-milliseconds test_16801_g17 100000
sentinel failover-timeout test_16801_g17 60000
sentinel client-reconfig-script test_16801_g17 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g18 192.168.11.11 6818 2
sentinel down-after-milliseconds test_16801_g18 100000
sentinel failover-timeout test_16801_g18 60000
sentinel client-reconfig-script test_16801_g18 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g19 192.168.11.11 6819 2
sentinel down-after-milliseconds test_16801_g19 100000
sentinel failover-timeout test_16801_g19 60000
sentinel client-reconfig-script test_16801_g19 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g20 192.168.11.11 6820 2
sentinel down-after-milliseconds test_16801_g20 100000
sentinel failover-timeout test_16801_g20 60000
sentinel client-reconfig-script test_16801_g20 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g21 192.168.11.12 6801 2
sentinel down-after-milliseconds test_16801_g21 100000
sentinel failover-timeout test_16801_g21 60000
sentinel client-reconfig-script test_16801_g21 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g22 192.168.11.12 6802 2
sentinel down-after-milliseconds test_16801_g22 100000
sentinel failover-timeout test_16801_g22 60000
sentinel client-reconfig-script test_16801_g22 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g23 192.168.11.12 6803 2
sentinel down-after-milliseconds test_16801_g23 100000
sentinel failover-timeout test_16801_g23 60000
sentinel client-reconfig-script test_16801_g23 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g24 192.168.11.12 6804 2
sentinel down-after-milliseconds test_16801_g24 100000
sentinel failover-timeout test_16801_g24 60000
sentinel client-reconfig-script test_16801_g24 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g25 192.168.11.12 6805 2
sentinel down-after-milliseconds test_16801_g25 100000
sentinel failover-timeout test_16801_g25 60000
sentinel client-reconfig-script test_16801_g25 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g26 192.168.11.12 6806 2
sentinel down-after-milliseconds test_16801_g26 100000
sentinel failover-timeout test_16801_g26 60000
sentinel client-reconfig-script test_16801_g26 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g27 192.168.11.12 6807 2
sentinel down-after-milliseconds test_16801_g27 100000
sentinel failover-timeout test_16801_g27 60000
sentinel client-reconfig-script test_16801_g27 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g28 192.168.11.12 6808 2
sentinel down-after-milliseconds test_16801_g28 100000
sentinel failover-timeout test_16801_g28 60000
sentinel client-reconfig-script test_16801_g28 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g29 192.168.11.12 6809 2
sentinel down-after-milliseconds test_16801_g29 100000
sentinel failover-timeout test_16801_g29 60000
sentinel client-reconfig-script test_16801_g29 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g30 192.168.11.12 6810 2
sentinel down-after-milliseconds test_16801_g30 100000
sentinel failover-timeout test_16801_g30 60000
sentinel client-reconfig-script test_16801_g30 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g31 192.168.11.12 6811 2
sentinel down-after-milliseconds test_16801_g31 100000
sentinel failover-timeout test_16801_g31 60000
sentinel client-reconfig-script test_16801_g31 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g32 192.168.11.12 6812 2
sentinel down-after-milliseconds test_16801_g32 100000
sentinel failover-timeout test_16801_g32 60000
sentinel client-reconfig-script test_16801_g32 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g33 192.168.11.12 6813 2
sentinel down-after-milliseconds test_16801_g33 100000
sentinel failover-timeout test_16801_g33 60000
sentinel client-reconfig-script test_16801_g33 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g34 192.168.11.12 6814 2
sentinel down-after-milliseconds test_16801_g34 100000
sentinel failover-timeout test_16801_g34 60000
sentinel client-reconfig-script test_16801_g34 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g35 192.168.11.12 6815 2
sentinel down-after-milliseconds test_16801_g35 100000
sentinel failover-timeout test_16801_g35 60000
sentinel client-reconfig-script test_16801_g35 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g36 192.168.11.12 6816 2
sentinel down-after-milliseconds test_16801_g36 100000
sentinel failover-timeout test_16801_g36 60000
sentinel client-reconfig-script test_16801_g36 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g37 192.168.11.12 6817 2
sentinel down-after-milliseconds test_16801_g37 100000
sentinel failover-timeout test_16801_g37 60000
sentinel client-reconfig-script test_16801_g37 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g38 192.168.11.12 6818 2
sentinel down-after-milliseconds test_16801_g38 100000
sentinel failover-timeout test_16801_g38 60000
sentinel client-reconfig-script test_16801_g38 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g39 192.168.11.12 6819 2
sentinel down-after-milliseconds test_16801_g39 100000
sentinel failover-timeout test_16801_g39 60000
sentinel client-reconfig-script test_16801_g39 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g40 192.168.11.12 6820 2
sentinel down-after-milliseconds test_16801_g40 100000
sentinel failover-timeout test_16801_g40 60000
sentinel client-reconfig-script test_16801_g40 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g41 192.168.11.13 6801 2
sentinel down-after-milliseconds test_16801_g41 100000
sentinel failover-timeout test_16801_g41 60000
sentinel client-reconfig-script test_16801_g41 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g42 192.168.11.13 6802 2
sentinel down-after-milliseconds test_16801_g42 100000
sentinel failover-timeout test_16801_g42 60000
sentinel client-reconfig-script test_16801_g42 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g43 192.168.11.13 6803 2
sentinel down-after-milliseconds test_16801_g43 100000
sentinel failover-timeout test_16801_g43 60000
sentinel client-reconfig-script test_16801_g43 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g44 192.168.11.13 6804 2
sentinel down-after-milliseconds test_16801_g44 100000
sentinel failover-timeout test_16801_g44 60000
sentinel client-reconfig-script test_16801_g44 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g45 192.168.11.13 6805 2
sentinel down-after-milliseconds test_16801_g45 100000
sentinel failover-timeout test_16801_g45 60000
sentinel client-reconfig-script test_16801_g45 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g46 192.168.11.13 6806 2
sentinel down-after-milliseconds test_16801_g46 100000
sentinel failover-timeout test_16801_g46 60000
sentinel client-reconfig-script test_16801_g46 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g47 192.168.11.13 6807 2
sentinel down-after-milliseconds test_16801_g47 100000
sentinel failover-timeout test_16801_g47 60000
sentinel client-reconfig-script test_16801_g47 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g48 192.168.11.13 6808 2
sentinel down-after-milliseconds test_16801_g48 100000
sentinel failover-timeout test_16801_g48 60000
sentinel client-reconfig-script test_16801_g48 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g49 192.168.11.13 6809 2
sentinel down-after-milliseconds test_16801_g49 100000
sentinel failover-timeout test_16801_g49 60000
sentinel client-reconfig-script test_16801_g49 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g50 192.168.11.13 6810 2
sentinel down-after-milliseconds test_16801_g50 100000
sentinel failover-timeout test_16801_g50 60000
sentinel client-reconfig-script test_16801_g50 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g51 192.168.11.13 6811 2
sentinel down-after-milliseconds test_16801_g51 100000
sentinel failover-timeout test_16801_g51 60000
sentinel client-reconfig-script test_16801_g51 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g52 192.168.11.13 6812 2
sentinel down-after-milliseconds test_16801_g52 100000
sentinel failover-timeout test_16801_g52 60000
sentinel client-reconfig-script test_16801_g52 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g53 192.168.11.13 6813 2
sentinel down-after-milliseconds test_16801_g53 100000
sentinel failover-timeout test_16801_g53 60000
sentinel client-reconfig-script test_16801_g53 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g54 192.168.11.13 6814 2
sentinel down-after-milliseconds test_16801_g54 100000
sentinel failover-timeout test_16801_g54 60000
sentinel client-reconfig-script test_16801_g54 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g55 192.168.11.13 6815 2
sentinel down-after-milliseconds test_16801_g55 100000
sentinel failover-timeout test_16801_g55 60000
sentinel client-reconfig-script test_16801_g55 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g56 192.168.11.13 6816 2
sentinel down-after-milliseconds test_16801_g56 100000
sentinel failover-timeout test_16801_g56 60000
sentinel client-reconfig-script test_16801_g56 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g57 192.168.11.13 6817 2
sentinel down-after-milliseconds test_16801_g57 100000
sentinel failover-timeout test_16801_g57 60000
sentinel client-reconfig-script test_16801_g57 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g58 192.168.11.13 6818 2
sentinel down-after-milliseconds test_16801_g58 100000
sentinel failover-timeout test_16801_g58 60000
sentinel client-reconfig-script test_16801_g58 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g59 192.168.11.13 6819 2
sentinel down-after-milliseconds test_16801_g59 100000
sentinel failover-timeout test_16801_g59 60000
sentinel client-reconfig-script test_16801_g59 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g60 192.168.11.13 6820 2
sentinel down-after-milliseconds test_16801_g60 100000
sentinel failover-timeout test_16801_g60 60000
sentinel client-reconfig-script test_16801_g60 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g68 192.168.11.14 6801 2
sentinel down-after-milliseconds test_16801_g68 100000
sentinel failover-timeout test_16801_g68 60000
sentinel client-reconfig-script test_16801_g68 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g62 192.168.11.14 6802 2
sentinel down-after-milliseconds test_16801_g62 100000
sentinel failover-timeout test_16801_g62 60000
sentinel client-reconfig-script test_16801_g62 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g63 192.168.11.14 6803 2
sentinel down-after-milliseconds test_16801_g63 100000
sentinel failover-timeout test_16801_g63 60000
sentinel client-reconfig-script test_16801_g63 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g64 192.168.11.14 6804 2
sentinel down-after-milliseconds test_16801_g64 100000
sentinel failover-timeout test_16801_g64 60000
sentinel client-reconfig-script test_16801_g64 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g65 192.168.11.14 6805 2
sentinel down-after-milliseconds test_16801_g65 100000
sentinel failover-timeout test_16801_g65 60000
sentinel client-reconfig-script test_16801_g65 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g66 192.168.11.14 6806 2
sentinel down-after-milliseconds test_16801_g66 100000
sentinel failover-timeout test_16801_g66 60000
sentinel client-reconfig-script test_16801_g66 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g67 192.168.11.14 6807 2
sentinel down-after-milliseconds test_16801_g67 100000
sentinel failover-timeout test_16801_g67 60000
sentinel client-reconfig-script test_16801_g67 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g68 192.168.11.14 6808 2
sentinel down-after-milliseconds test_16801_g68 100000
sentinel failover-timeout test_16801_g68 60000
sentinel client-reconfig-script test_16801_g68 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g69 192.168.11.14 6809 2
sentinel down-after-milliseconds test_16801_g69 100000
sentinel failover-timeout test_16801_g69 60000
sentinel client-reconfig-script test_16801_g69 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g70 192.168.11.14 6810 2
sentinel down-after-milliseconds test_16801_g70 100000
sentinel failover-timeout test_16801_g70 60000
sentinel client-reconfig-script test_16801_g70 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g71 192.168.11.14 6811 2
sentinel down-after-milliseconds test_16801_g71 100000
sentinel failover-timeout test_16801_g71 60000
sentinel client-reconfig-script test_16801_g71 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g72 192.168.11.14 6812 2
sentinel down-after-milliseconds test_16801_g72 100000
sentinel failover-timeout test_16801_g72 60000
sentinel client-reconfig-script test_16801_g72 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g73 192.168.11.14 6813 2
sentinel down-after-milliseconds test_16801_g73 100000
sentinel failover-timeout test_16801_g73 60000
sentinel client-reconfig-script test_16801_g73 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g74 192.168.11.14 6814 2
sentinel down-after-milliseconds test_16801_g74 100000
sentinel failover-timeout test_16801_g74 60000
sentinel client-reconfig-script test_16801_g74 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g75 192.168.11.14 6815 2
sentinel down-after-milliseconds test_16801_g75 100000
sentinel failover-timeout test_16801_g75 60000
sentinel client-reconfig-script test_16801_g75 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g76 192.168.11.14 6816 2
sentinel down-after-milliseconds test_16801_g76 100000
sentinel failover-timeout test_16801_g76 60000
sentinel client-reconfig-script test_16801_g76 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g77 192.168.11.14 6817 2
sentinel down-after-milliseconds test_16801_g77 100000
sentinel failover-timeout test_16801_g77 60000
sentinel client-reconfig-script test_16801_g77 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g78 192.168.11.14 6818 2
sentinel down-after-milliseconds test_16801_g78 100000
sentinel failover-timeout test_16801_g78 60000
sentinel client-reconfig-script test_16801_g78 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g79 192.168.11.14 6819 2
sentinel down-after-milliseconds test_16801_g79 100000
sentinel failover-timeout test_16801_g79 60000
sentinel client-reconfig-script test_16801_g79 /etc/sentinel/client-reconfig.sh
sentinel monitor test_16801_g80 192.168.11.14 6820 2
sentinel down-after-milliseconds test_16801_g80 100000
sentinel failover-timeout test_16801_g80 60000
sentinel client-reconfig-script test_16801_g80 /etc/sentinel/client-reconfig.sh
5.2. 编辑Redis主从切换执行脚本
# vim /etc/sentinel/client-reconfig.sh
#!/bin/sh
### <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
monitor_name="$1"
master_old_ip="$4"
master_old_port="$5"
master_new_ip="$6"
master_new_port="$7"
twemproxy_base="/test/nutcracker/nutcracker/bin"
twemproxy_bin="${twemproxy_base}/nutcracker"
twemproxy_conf="${twemproxy_base}/conf/nutcracker.yml"
time=`date '+%Y-%m-%d %H:%M:%S'`
echo "[${time}] ${master_old_ip}:${master_old_port} start failover!" >> /var/log/redis_failover.log
# 将新的master 端口和ip 替换掉 twemproxy 配置文件中旧的master 信息
flock -e -w 10 ${twemproxy_conf} -c "sed -i \"s/${master_old_ip}:${master_old_port}/${master_new_ip}:${master_new_port}/\" ${twemproxy_conf}"
# 如果修改成功,重启twemproxy,否则记录失败日志
if [ $? -eq 0 ]; then
killall spawn_twemproxy
killall nutcracker
sleep 1
cd ${twemproxy_base}
bash ./start.sh
time=`date '+%Y-%m-%d %H:%M:%S'`
echo "[${time}] ${master_old_ip}:${master_old_port} failover to ${master_new_ip}:${master_new_port} ok!" >> /var/log/redis_failover.log
else
time=`date '+%Y-%m-%d %H:%M:%S'`
echo "[${time}] ${master_old_ip}:${master_old_port} failover to ${master_new_ip}:${master_new_port} failed!" >> /var/log/redis_failover.log
fi
为Redis主从切换脚本赋予执行权限:
# chmod +x /etc/sentinel/client-reconfig.sh
5.3. 启动哨兵
启动哨兵:
# cd /etc/sentinel/
# redis-sentinel /etc/sentinel/sentinel.conf
查看进程:
# ps -ef|grep redis-sentinel|grep -v grep
5.4. 查看哨兵集群
# redis-cli -p 26880 info sentinel
# Sentinel
sentinel_masters:80
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=test_16801_g56,status=ok,address=192.168.11.13:6816,slaves=1,sentinels=4
master1:name=test_16801_g27,status=ok,address=192.168.11.12:6807,slaves=1,sentinels=4
master2:name=test_16801_g75,status=ok,address=192.168.11.14:6815,slaves=1,sentinels=4
master3:name=test_16801_g49,status=ok,address=192.168.11.13:6809,slaves=1,sentinels=4
master4:name=test_16801_g22,status=ok,address=192.168.11.12:6802,slaves=1,sentinels=4
master5:name=test_16801_g18,status=ok,address=192.168.11.11:6818,slaves=1,sentinels=4
master6:name=test_16801_g41,status=ok,address=192.168.11.13:6801,slaves=1,sentinels=4
master7:name=test_16801_g50,status=ok,address=192.168.11.13:6810,slaves=1,sentinels=4
master8:name=test_16801_g47,status=ok,address=192.168.11.13:6807,slaves=1,sentinels=4
master9:name=test_16801_g21,status=ok,address=192.168.11.12:6801,slaves=1,sentinels=4
master10:name=test_16801_g62,status=ok,address=192.168.11.14:6802,slaves=1,sentinels=4
master11:name=test_16801_g64,status=ok,address=192.168.11.14:6804,slaves=1,sentinels=4
master12:name=test_16801_g5,status=ok,address=192.168.11.11:6805,slaves=1,sentinels=4
master13:name=test_16801_g15,status=ok,address=192.168.11.11:6815,slaves=1,sentinels=4
master14:name=test_16801_g12,status=ok,address=192.168.11.11:6812,slaves=1,sentinels=4
master15:name=test_16801_g11,status=ok,address=192.168.11.11:6811,slaves=1,sentinels=4
master16:name=test_16801_g16,status=ok,address=192.168.11.11:6816,slaves=1,sentinels=4
master17:name=test_16801_g48,status=ok,address=192.168.11.13:6808,slaves=1,sentinels=4
master18:name=test_16801_g14,status=ok,address=192.168.11.11:6814,slaves=1,sentinels=4
master19:name=test_16801_g13,status=ok,address=192.168.11.11:6813,slaves=1,sentinels=4
master20:name=test_16801_g80,status=ok,address=192.168.11.14:6820,slaves=1,sentinels=4
master21:name=test_16801_g58,status=ok,address=192.168.11.13:6818,slaves=1,sentinels=4
master22:name=test_16801_g67,status=ok,address=192.168.11.14:6807,slaves=1,sentinels=4
master23:name=test_16801_g45,status=ok,address=192.168.11.13:6805,slaves=1,sentinels=4
master24:name=test_16801_g37,status=ok,address=192.168.11.12:6817,slaves=1,sentinels=4
master25:name=test_16801_g34,status=ok,address=192.168.11.12:6814,slaves=1,sentinels=4
master26:name=test_16801_g17,status=ok,address=192.168.11.11:6817,slaves=1,sentinels=4
master27:name=test_16801_g42,status=ok,address=192.168.11.13:6802,slaves=1,sentinels=4
master28:name=test_16801_g31,status=ok,address=192.168.11.12:6811,slaves=1,sentinels=4
master29:name=test_16801_g79,status=ok,address=192.168.11.14:6819,slaves=1,sentinels=4
master30:name=test_16801_g54,status=ok,address=192.168.11.13:6814,slaves=1,sentinels=4
master31:name=test_16801_g53,status=ok,address=192.168.11.13:6813,slaves=1,sentinels=4
master32:name=test_16801_g46,status=ok,address=192.168.11.13:6806,slaves=1,sentinels=4
master33:name=test_16801_g71,status=ok,address=192.168.11.14:6811,slaves=1,sentinels=4
master34:name=test_16801_g1,status=ok,address=192.168.11.11:6801,slaves=1,sentinels=4
master35:name=test_16801_g3,status=ok,address=192.168.11.11:6803,slaves=1,sentinels=4
master36:name=test_16801_g77,status=ok,address=192.168.11.14:6817,slaves=1,sentinels=4
master37:name=test_16801_g74,status=ok,address=192.168.11.14:6814,slaves=1,sentinels=4
master38:name=test_16801_g33,status=ok,address=192.168.11.12:6813,slaves=1,sentinels=4
master39:name=test_16801_g59,status=ok,address=192.168.11.13:6819,slaves=1,sentinels=4
master40:name=test_16801_g23,status=ok,address=192.168.11.12:6803,slaves=1,sentinels=4
master41:name=test_16801_g68,status=ok,address=192.168.11.14:6808,slaves=1,sentinels=4
master42:name=test_16801_g9,status=ok,address=192.168.11.11:6809,slaves=1,sentinels=4
master43:name=test_16801_g25,status=ok,address=192.168.11.12:6805,slaves=1,sentinels=4
master44:name=test_16801_g36,status=ok,address=192.168.11.12:6816,slaves=1,sentinels=4
master45:name=test_16801_g26,status=ok,address=192.168.11.12:6806,slaves=1,sentinels=4
master46:name=test_16801_g66,status=ok,address=192.168.11.14:6806,slaves=1,sentinels=4
master47:name=test_16801_g78,status=ok,address=192.168.11.14:6818,slaves=1,sentinels=4
master48:name=test_16801_g20,status=ok,address=192.168.11.11:6820,slaves=1,sentinels=4
master49:name=test_16801_g52,status=ok,address=192.168.11.13:6812,slaves=1,sentinels=4
master50:name=test_16801_g24,status=ok,address=192.168.11.12:6804,slaves=1,sentinels=4
master51:name=test_16801_g51,status=ok,address=192.168.11.13:6811,slaves=1,sentinels=4
master52:name=test_16801_g60,status=ok,address=192.168.11.13:6820,slaves=1,sentinels=4
master53:name=test_16801_g35,status=ok,address=192.168.11.12:6815,slaves=1,sentinels=4
master54:name=test_16801_g10,status=ok,address=192.168.11.11:6810,slaves=1,sentinels=4
master55:name=test_16801_g70,status=ok,address=192.168.11.14:6810,slaves=1,sentinels=4
master56:name=test_16801_g63,status=ok,address=192.168.11.14:6803,slaves=1,sentinels=4
master57:name=test_16801_g76,status=ok,address=192.168.11.14:6816,slaves=1,sentinels=4
master58:name=test_16801_g30,status=ok,address=192.168.11.12:6810,slaves=1,sentinels=4
master59:name=test_16801_g6,status=ok,address=192.168.11.11:6806,slaves=1,sentinels=4
master60:name=test_16801_g38,status=ok,address=192.168.11.12:6818,slaves=1,sentinels=4
master68:name=test_16801_g19,status=ok,address=192.168.11.11:6819,slaves=1,sentinels=4
master62:name=test_16801_g68,status=ok,address=192.168.11.14:6801,slaves=1,sentinels=4
master63:name=test_16801_g39,status=ok,address=192.168.11.12:6819,slaves=1,sentinels=4
master64:name=test_16801_g2,status=ok,address=192.168.11.11:6802,slaves=1,sentinels=4
master65:name=test_16801_g40,status=ok,address=192.168.11.12:6820,slaves=1,sentinels=4
master66:name=test_16801_g55,status=ok,address=192.168.11.13:6815,slaves=1,sentinels=4
master67:name=test_16801_g28,status=ok,address=192.168.11.12:6808,slaves=1,sentinels=4
master68:name=test_16801_g57,status=ok,address=192.168.11.13:6817,slaves=1,sentinels=4
master69:name=test_16801_g43,status=ok,address=192.168.11.13:6803,slaves=1,sentinels=4
master70:name=test_16801_g65,status=ok,address=192.168.11.14:6805,slaves=1,sentinels=4
master71:name=test_16801_g44,status=ok,address=192.168.11.13:6804,slaves=1,sentinels=4
master72:name=test_16801_g73,status=ok,address=192.168.11.14:6813,slaves=1,sentinels=4
master73:name=test_16801_g32,status=ok,address=192.168.11.12:6812,slaves=1,sentinels=4
master74:name=test_16801_g7,status=ok,address=192.168.11.11:6807,slaves=1,sentinels=4
master75:name=test_16801_g72,status=ok,address=192.168.11.14:6812,slaves=1,sentinels=4
master76:name=test_16801_g29,status=ok,address=192.168.11.12:6809,slaves=1,sentinels=4
master77:name=test_16801_g8,status=ok,address=192.168.11.11:6808,slaves=1,sentinels=4
master78:name=test_16801_g4,status=ok,address=192.168.11.11:6804,slaves=1,sentinels=4
master79:name=test_16801_g69,status=ok,address=192.168.11.14:6809,slaves=1,sentinels=4
6. 验证twemproxy集群
6.1. 验证twemproxy集群的可用性
# redis-cli -p 16801 set testTwemproxy Twemproxy
OK
# redis-cli -p 16801 get testTwemproxy
"Twemproxy"
验证成功,可正常读写。
6.2. 验证Redis主库宕机后twemproxy集群可正常使用
6.2.1. 查看 Redis 复制集群
如下,192.168.11.11:6801和192.168.11.14:6821 为一对Redis主从:
# redis-cli -h 192.168.11.11 -p 6801 role
1) "master"
2) (integer) 28343
3) 1) 1) "192.168.11.14"
2) "6821"
3) "27905"
# redis-cli -h 192.168.11.14 -p 6821 role
1) "slave"
2) "192.168.11.11"
3) (integer) 6801
4) "connected"
5) (integer) 31827
6.2.2. 停止复制集群的192.168.11.11:6801主库节点
/* 192.168.11.11:6801 */
停止复制集群的192.168.11.11:6801主库节点:
# ps -ef| grep redis-server|grep 6801| grep -v grep
# redis-cli -h 192.168.11.11 -p 6801 shutdown
确认节点已经停止:
# ps -ef| grep redis-server|grep 6801| grep -v grep
6.2.3. 查看监控日志
查看哨兵的监控日志:
# tail -f /var/log/sentinel.log
104017:X 06 Nov 12:45:18.118 * +sentinel sentinel 3cc080c86cd407b38ec8797e2aef12c19b771bc2 192.168.11.11 26880 @ test_16801_g29 192.168.11.12 6809
104017:X 06 Nov 12:45:18.121 * +sentinel sentinel 3cc080c86cd407b38ec8797e2aef12c19b771bc2 192.168.11.11 26880 @ test_16801_g55 192.168.11.13 6815
104017:X 06 Nov 12:47:43.820 # +sdown master test_16801_g1 192.168.11.11 6801
104017:X 06 Nov 12:47:43.868 # +new-epoch 1
104017:X 06 Nov 12:47:43.865 # +vote-for-leader ea5e6485ee7c4b6b0c52da29001c2fc868389712 1
104017:X 06 Nov 12:47:43.910 # +odown master test_16801_g1 192.168.11.11 6801 #quorum 4/2
104017:X 06 Nov 12:47:43.910 # Next failover delay: I will not start a failover before Fri Nov 6 12:49:43 2020
104017:X 06 Nov 12:47:44.586 # +config-update-from sentinel ea5e6485ee7c4b6b0c52da29001c2fc868389712 192.168.11.14 26880 @ test_16801_g1 192.168.11.11 6801
104017:X 06 Nov 12:47:44.586 # +switch-master test_16801_g1 192.168.11.11 6801 192.168.11.14 6821
104017:X 06 Nov 12:47:44.586 * +slave slave 192.168.11.11:6801 192.168.11.11 6801 @ test_16801_g1 192.168.11.14 6821
104017:X 06 Nov 12:47:54.660 # +sdown slave 192.168.11.11:6801 192.168.11.11 6801 @ test_16801_g1 192.168.11.14 6821
6.2.4. 查看master test_16801_g1 的信息
从哨兵配置文件中可以看出,监控的master test_16801_g1发生了主从切换:
# grep -E 'sentinel known-slave test_16801_g1|sentinel monitor test_16801_g1' /etc/sentinel/sentinel.conf
sentinel monitor test_16801_g1 192.168.11.14 6821 2
sentinel known-slave test_16801_g1 192.168.11.11 6801
从哨兵查看mater test_16801_g1的信息:
# redis-cli -p 26880
127.0.0.1:26880> sentinel master test_16801_g1
1) "name"
2) "test_16801_g1"
3) "ip"
4) "192.168.11.14"
5) "port"
6) "6821"
7) "runid"
8) "0376a9459c562a9443d801023535ef436a1a4552"
9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "479"
19) "last-ping-reply"
20) "479"
21) "down-after-milliseconds"
22) "10000"
23) "info-refresh"
24) "5767"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "66669"
29) "config-epoch"
30) "1"
31) "num-slaves"
32) "1"
33) "num-other-sentinels"
34) "3"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "60000"
39) "parallel-syncs"
40) "1"
41) "client-reconfig-script"
42) "/etc/sentinel/client-reconfig.sh"
6.2.5. 查看twemproxy的配置信息,确认切换脚本成功切换
Redis主从切换,引发脚本/etc/sentinel/client-reconfig.sh更新了/nutcracker.yml配置文件并重启:
# cat /test/nutcracker/nutcracker/bin/conf/nutcracker.yml | grep 192.168.11.11 | grep 6801
# cat /test/nutcracker/nutcracker/bin/conf/nutcracker.yml | grep 192.168.11.14 | grep 6821
- 192.168.11.14:6821:1 server1
6.2.6. 启动宕机的Redis节点
/* 192.168.11.11:6801 */
启动宕机的Redis节点:
# cd /data/redis/16801/etc/
# redis-server /data/redis/16801/etc/6801.conf
查看进程:
# ps -ef| grep redis-server|grep 6801|grep -v grep
查看Redis节点角色:
# redis-cli -h 192.168.11.11 -p 6801 role
1) "slave"
2) "192.168.11.14"
3) (integer) 6821
4) "connected"
5) (integer) 86581
# redis-cli -h 192.168.11.14 -p 6821 role
1) "master"
2) (integer) 89668
3) 1) 1) "192.168.11.11"
2) "6801"
3) "89515"
6.2.7. 手动进行主从切换,恢复原主从复制关系
# redis-cli -p 26880
127.0.0.1:26880> sentinel failover test_16801_g1
OK
127.0.0.1:26880> sentinel master test_16801_g1
1) "name"
2) "test_16801_g1"
3) "ip"
4) "192.168.11.11"
5) "port"
6) "6801"
7) "runid"
8) ""
9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "956"
19) "last-ping-reply"
20) "956"
21) "down-after-milliseconds"
22) "10000"
23) "info-refresh"
24) "5131"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "2102"
29) "config-epoch"
30) "2"
31) "num-slaves"
32) "1"
33) "num-other-sentinels"
34) "3"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "60000"
39) "parallel-syncs"
40) "1"
41) "client-reconfig-script"
42) "/etc/sentinel/client-reconfig.sh"
查看哨兵配置文件,确认原复制关系已经恢复:
# grep -E 'sentinel known-slave test_16801_g1|sentinel monitor test_16801_g1' /etc/sentinel/sentinel.conf
sentinel monitor test_16801_g1 192.168.11.11 6801 2
sentinel known-slave test_16801_g1 192.168.11.14 6821
# redis-cli -p 26880
1) "name"
2) "test_16801_g1"
3) "ip"
4) "192.168.11.11"
5) "port"
6) "6801"
7) "runid"
8) "645ff55a5590e5e280286790a5939c6b3da5f7bd"
9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "919"
19) "last-ping-reply"
20) "919"
21) "down-after-milliseconds"
22) "10000"
23) "info-refresh"
24) "5867"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "42627"
29) "config-epoch"
30) "2"
31) "num-slaves"
32) "1"
33) "num-other-sentinels"
34) "3"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "60000"
39) "parallel-syncs"
40) "1"
41) "client-reconfig-script"
42) "/etc/sentinel/client-reconfig.sh"
查看twemproxy,确认nutcracker已经恢复最初的配置并完成了重启:
# cat /test/nutcracker/nutcracker/bin/conf/nutcracker.yml | grep 192.168.11.11 | grep 6801
- 192.168.11.11:6801:1 server1
# cat /test/nutcracker/nutcracker/bin/conf/nutcracker.yml | grep 192.168.11.14 | grep 6821