基础环境准备
两台服务器:192.168.199.101(主)、192.168.199.102(从),每台服务器上分别安装nginx 、keepalived。
安装nginx:
yum -y install nginx
启动nginx:
systemctl start nginx
安装keepalived:
yum -y install keepalived
启动keepalived:
systemctl start keepalived
主从配置:
节点规划
主机名 | ip | 软件 | 端口 | 角色 |
---|---|---|---|---|
yfm01 | 192.168.199.101 | nginx、keepalived | 80 | 主节点 |
yfm02 | 192.168.199.102 | nginx、keepalived | 80 | 从节点 |
VIP | 192.168.199.99 |
设置keepalived日志单独存放
默认keepalived日志存放在/var/log/messages
sed -i s#KEEPALIVED_OPTIONS=\"-D\"#KEEPALIVED_OPTIONS=\"-D\ -d\ -S\ 0\"#g /etc/sysconfig/keepalived
echo "local0.* /var/log/keepalived.log" >> /etc/rsyslog.conf
修改/lib/systemd/system/keepalived.service
sed -i s#EnvironmentFile=-/etc/sysconfig/keepalived#EnvironmentFile=/etc/sysconfig/keepalived#g /lib/systemd/system/keepalived.service
重新加载service:systemctl daemon-reload #这种我试了不行
重启机器reboot
主192.168.199.101,/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.199.101 #指定smtp服务器地址
smtp_connect_timeout 30
router_id LVS_DEVEL1 #负载均衡标识,在局域网内应该是唯一的
#vrrp_strict 不要开启
}
vrrp_script chk_http_port {
script "/etc/keepalived/nginx_check.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER #指定该keepalived节点的初始状态
interface ens33
virtual_router_id 51 #指定VRRP实例ID
priority 100 #指定优先级,优先级高的将成为MASTER
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.99 #指定VIP地址
}
}
从192.168.199.102,/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.199.101
smtp_connect_timeout 30
router_id LVS_DEVEL2
}
vrrp_script chk_http_port {
script "/etc/keepalived/nginx_check.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.99
}
}
nginx_check.sh
#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
/usr/sbin/nginx
sleep 2
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
systemctl stop keepalived
fi
fi
总结:
- master没挂,则master占有vip且nginx运行在master上
- master挂了,则slave抢占vip且在slave上运行nginx服务
- 如果master上的nginx服务挂了,则nginx会自动重启,重启失败后会自动关闭keepalived,这样vip资源也会转移到slave上。
- master和slave两边都开启nginx服务,无论master还是slave,当其中的一个keepalived服务停止后,vip都会漂移到keepalived服务还在的节点上;
- 如果要想使nginx服务挂了,vip也漂移到另一个节点,则必须用脚本或者在配置文件里面用shell命令来控制。(nginx服务宕停后会自动启动,启动失败后会强制关闭keepalived,从而致使vip资源漂移到另一台机器上)
主从的缺点在于如果主机一直稳定,那么从机就会一直处于空闲状态,造成了资源的浪费。
主主配置
节点规划
主机名 | ip | 软件 | 端口 | 角色 |
---|---|---|---|---|
yfm01 | 192.168.199.101 | nginx、keepalived | 80 | 主、从节点 |
yfm02 | 192.168.199.102 | nginx、keepalived | 80 | 从、从节点 |
VIP | 192.168.199.99、192.168.199.98 |
192.168.199.101
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.199.101 #指定smtp服务器地址
smtp_connect_timeout 30
router_id LVS_DEVEL1 #负载均衡标识,在局域网内应该是唯一的
}
vrrp_script chk_http_port {
script "/etc/keepalived/nginx_check.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER #指定该keepalived节点的初始状态
interface ens33
virtual_router_id 51 #指定VRRP实例ID
priority 100 #指定优先级,优先级高的将成为MASTER
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.99 #指定VIP地址
}
track_script {
chk_http_port
}
}
vrrp_instance VI_2 {
state BACKUP #这里修改为BACKUP
interface ens33
virtual_router_id 52 #这里修改为52
priority 90 #修改优先级
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.98 #绑定另外一个VIP
}
track_script {
chk_http_port
}
}
192.168.199.102
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.199.101
smtp_connect_timeout 30
router_id LVS_DEVEL2
}
vrrp_script chk_http_port {
script "/etc/keepalived/nginx_check.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.99
}
track_script {
chk_http_port
}
}
vrrp_instance VI_2 {
state MASTER
interface ens33
virtual_router_id 52
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.98
}
track_script {
chk_http_port
}
}
nginx_check.sh
#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
/usr/sbin/nginx
sleep 2
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
systemctl stop keepalived
fi
fi
对VRRP_Instance(VI_1)而言,192.168.199.101为MASTER;对VRRP_Instance(VI_2)而言,192.168.199.102为MASTER。
浏览器访问http://192.168.199.99/,192.168.199.101上的nginx会处理该请求;浏览器访问http://192.168.199.98/,192.168.199.102上的nginx会处理该请求 !
当任何一台服务器上面的keepalived服务停掉后,另外一台服务器上面的keepalived都会进入MASTER状态处理请求。停掉192.168.199.101上的keepalived服务,则192.168.199.102上的keepalived服务的VRRP_Instance(VI_1)进入MASTER状态,并对http://192.168.199.99/和http://192.168.199.98/进行处理!
访问http://192.168.199.99/
访问http://192.168.199.98/
lvs + keepalived + nginx 主从模式
节点规划
主机名 | ip | 软件 | 端口 | 角色 |
---|---|---|---|---|
yfm01 | 192.168.199.101 | lvs、keepalived | 80 | MASTER |
yfm02 | 192.168.199.102 | lvs、keepalived | 80 | SLAVE |
yfm03 | 192.168.199.103 | nginx | 80 | realserver1 |
yfm04 | 192.168.199.104 | nginx | 80 | realserver2 |
VIP | 192.168.199.99 |
在yfm01、yfm02节点安装lvs
yum -y install ipvsadm
加载ipvsadm模块进系统
ipvsadm
在yfm03、yfm04节点安装nginx
yum -y install nginx
192.168.199.101、/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_Master
}
vrrp_instance VI_1 {
state MASTER #指定instance初始状态,实际根据优先级决定.backup节点不一样
interface ens33 #虚拟IP所在网
virtual_router_id 51 #VRID,相同VRID为一个组,决定多播MAC地址
priority 100 #优先级,另一台改为90.backup节点不一样
advert_int 1 #检查间隔
authentication {
auth_type PASS #认证方式,可以是pass或ha
auth_pass 1111 #认证密码
}
virtual_ipaddress {
192.168.199.99 #VIP
}
}
virtual_server 192.168.199.99 80 {
delay_loop 6 #服务轮询的时间间隔
lb_algo wrr #加权轮询调度,LVS调度算法 rr|wrr|lc|wlc|lblc|sh|sh
lb_kind DR #LVS集群模式 NAT|DR|TUN,其中DR模式要求负载均衡器网卡必须有一块与物理网卡在同一个网段
#nat_mask 255.255.255.0
persistence_timeout 50 #会话保持时间
protocol TCP #健康检查协议
## Real Server设置,80就是MySQL连接端口
real_server 192.168.199.103 80 {
weight 3 ##权重
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.199.104 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
192.168.199.102、/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_Backup
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.99
}
}
virtual_server 192.168.199.99 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.199.103 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.199.104 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
192.168.199.103、192.168.199.104
新建/etc/init.d/realserver脚本
#!/bin/sh
#chkconfig: 2345 90 10
VIP=192.168.199.99
. /etc/rc.d/init.d/functions
case "$1" in
# 禁用本地的ARP请求、绑定本地回环地址
start)
/sbin/ifconfig lo down
/sbin/ifconfig lo up
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
/sbin/sysctl -p >/dev/null 2>&1
/sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up #在回环地址上绑定VIP,设定掩码,与Direct Server(自身)上的IP保持通信
/sbin/route add -host $VIP dev lo:0
echo "LVS-DR real server starts successfully.\n"
;;
stop)
/sbin/ifconfig lo:0 down
/sbin/route del $VIP >/dev/null 2>&1
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
;;
status)
isLoOn=`/sbin/ifconfig lo:0 | grep "$VIP"`
isRoOn=`/bin/netstat -rn | grep "$VIP"`
if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
echo "LVS-DR real server has run yet."
else
echo "LVS-DR real server is running."
fi
exit 3
;;
*)
echo "Usage: $0 {start|stop|status}"
exit 1
esac
exit 0
赋予可执行权限
chomd +x realserver
开机启动
chkconfig realserver on
使用ip addr命令查看yfm01、yfm02,可以看到yfm01上绑定了vip,yfm02没有
在yfm01上执行ipvsadm -ln检查连接情况
修改yfm03、yfm04,nginx欢迎界面,添加上本机ip
whereis nginx
[root@yfm03 nginx]# whereis nginx
nginx: /usr/sbin/nginx /usr/lib64/nginx /etc/nginx /usr/share/nginx /usr/share/man/man3/nginx.3pm.gz /usr/share/man/man8/nginx.8.gz
可以看到nginx配置文件在 /etc/nginx目录
在浏览器访问192.168.199.103、192.168.199.104、192.168.199.99均可访问nginx服务
总结:
- 当 MASTER 服务器无法提供服务时,VIP 会在 MASTER 上自动移除,BACKUP 服务器会提升为 MASTER 状态,绑定 VIP 、接管服务。
- 当 MASTER 修复加入网络后,会自动抢回 VIP ,成为 MASTER 身份。
- 当后端提供服务nginx服务挂起时,会自动切换至其它nginx服务器。
lvs + keepalived + nginx 主主模式
主主环境相比于主从环境,区别只在于
- LVS负载均衡层需要两个VIP。比如192.168.199.98和192.168.199.99
- 后端的realserver上要绑定这两个VIP到lo本地回环口上
- Keepalived.conf的配置相比于上面的主从模式也有所不同
在yfm03,/etc/init.d目录新建realserver98、realserver99
realserver98
#!/bin/sh
#chkconfig: 2345 90 10
VIP=192.168.199.98
. /etc/rc.d/init.d/functions
case "$1" in
start)
/sbin/ifconfig lo down
/sbin/ifconfig lo up
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
/sbin/sysctl -p >/dev/null 2>&1
/sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
echo "LVS-DR real server starts successfully.\n"
;;
stop)
/sbin/ifconfig lo:0 down
/sbin/route del $VIP >/dev/null 2>&1
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
;;
status)
isLoOn=`/sbin/ifconfig lo:0 | grep "$VIP"`
isRoOn=`/bin/netstat -rn | grep "$VIP"`
if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
echo "LVS-DR real server has run yet."
else
echo "LVS-DR real server is running."
fi
exit 3
;;
*)
echo "Usage: $0 {start|stop|status}"
exit 1
esac
exit 0
realserver99
#!/bin/sh
#chkconfig: 2345 90 10
VIP=192.168.199.99
. /etc/rc.d/init.d/functions
case "$1" in
start)
/sbin/ifconfig lo down
/sbin/ifconfig lo up
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
/sbin/sysctl -p >/dev/null 2>&1
/sbin/ifconfig lo:1 $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:1
echo "LVS-DR real server starts successfully.\n"
;;
stop)
/sbin/ifconfig lo:1 down
/sbin/route del $VIP >/dev/null 2>&1
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
;;
status)
isLoOn=`/sbin/ifconfig lo:1 | grep "$VIP"`
isRoOn=`/bin/netstat -rn | grep "$VIP"`
if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
echo "LVS-DR real server has run yet."
else
echo "LVS-DR real server is running."
fi
exit 3
;;
*)
echo "Usage: $0 {start|stop|status}"
exit 1
esac
exit 0
赋予可执行权限
chomd +x realserver9*
在yfm04执行以下命令,复制刚刚新建的文件
scp root@192.168.199.103:/etc/init.d/realserver9* /etc/init.d/
在yfm03、yfm04执行以下命令
service realserver99 start
service realserver98 start
可以发现两个VIP已经成功绑定到lo上了
yfm01,/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_Master
}
vrrp_instance VI_1 {
state MASTER #指定instance初始状态,实际根据优先级决定.backup节点不一样
interface ens33 #虚拟IP所在网
virtual_router_id 51 #VRID,相同VRID为一个组,决定多播MAC地址
priority 100 #优先级,另一台改为90.backup节点不一样
advert_int 1 #检查间隔
authentication {
auth_type PASS #认证方式,可以是pass或ha
auth_pass 1111 #认证密码
}
virtual_ipaddress {
192.168.199.99 #VIP
}
}
vrrp_instance VI_2 {
state BACKUP
interface ens33
virtual_router_id 52
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.98
}
}
virtual_server 192.168.199.99 80 {
delay_loop 6 #服务轮询的时间间隔
lb_algo wrr #加权轮询调度,LVS调度算法 rr|wrr|lc|wlc|lblc|sh|sh
lb_kind DR #LVS集群模式 NAT|DR|TUN,其中DR模式要求负载均衡器网卡必须有一块与物理网卡在同一个网段
#nat_mask 255.255.255.0
persistence_timeout 50 #会话保持时间
protocol TCP #健康检查协议
## Real Server设置,80就是MySQL连接端口
real_server 192.168.199.103 80 {
weight 3 ##权重
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.199.104 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
virtual_server 192.168.199.98 80 {
delay_loop 6 #服务轮询的时间间隔
lb_algo wrr #加权轮询调度,LVS调度算法 rr|wrr|lc|wlc|lblc|sh|sh
lb_kind DR #LVS集群模式 NAT|DR|TUN,其中DR模式要求负载均衡器网卡必须有一块与物理网卡在同一个网段
#nat_mask 255.255.255.0
persistence_timeout 50 #会话保持时间
protocol TCP #健康检查协议
## Real Server设置,80就是MySQL连接端口
real_server 192.168.199.103 80 {
weight 3 ##权重
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.199.104 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
yfm02,/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_Backup
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.199.99
}
}
vrrp_instance VI_2 {
state Master
interface ens33
virtual_router_id 52
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
182.148.15.235
}
}
virtual_server 192.168.199.99 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.199.103 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.199.104 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
virtual_server 192.168.199.98 80 {
delay_loop 6
lb_algo wrr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.199.103 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.199.104 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
在yfm01,打开ip_forward路由转发功能
echo "1" > /proc/sys/net/ipv4/ip_forward
在yfm01、yfm02上启动keepalived
service keepalived start
可以看到,yfm01已经成功绑定两个VIP,而yfm02没有绑定;