lvs+nginx+keepalived

基础环境准备

两台服务器:192.168.199.101(主)、192.168.199.102(从),每台服务器上分别安装nginx 、keepalived。

安装nginx:
yum -y install nginx
启动nginx:
systemctl start nginx

安装keepalived:
yum -y install keepalived
启动keepalived:
systemctl start keepalived
主从配置:

节点规划

主机名ip软件端口角色
yfm01192.168.199.101nginx、keepalived80主节点
yfm02192.168.199.102nginx、keepalived80从节点
VIP192.168.199.99

设置keepalived日志单独存放

默认keepalived日志存放在/var/log/messages

sed -i s#KEEPALIVED_OPTIONS=\"-D\"#KEEPALIVED_OPTIONS=\"-D\ -d\ -S\ 0\"#g /etc/sysconfig/keepalived

echo "local0.* /var/log/keepalived.log" >> /etc/rsyslog.conf

修改/lib/systemd/system/keepalived.service
sed -i s#EnvironmentFile=-/etc/sysconfig/keepalived#EnvironmentFile=/etc/sysconfig/keepalived#g /lib/systemd/system/keepalived.service

重新加载service:systemctl daemon-reload  #这种我试了不行
重启机器reboot

主192.168.199.101,/etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.199.101 #指定smtp服务器地址
   smtp_connect_timeout 30
   router_id LVS_DEVEL1 #负载均衡标识,在局域网内应该是唯一的
   #vrrp_strict 不要开启
}

vrrp_script chk_http_port {
        script "/etc/keepalived/nginx_check.sh"
        interval 2
        weight 2
}

vrrp_instance VI_1 {
    state MASTER #指定该keepalived节点的初始状态
    interface ens33
    virtual_router_id 51 #指定VRRP实例ID
    priority 100 #指定优先级,优先级高的将成为MASTER
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.99 #指定VIP地址
    }
}

从192.168.199.102,/etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.199.101
   smtp_connect_timeout 30
   router_id LVS_DEVEL2
}
vrrp_script chk_http_port {
        script "/etc/keepalived/nginx_check.sh"
        interval 2
        weight 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.99
    }
}

nginx_check.sh

#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
    /usr/sbin/nginx
    sleep 2
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        systemctl stop keepalived
    fi
fi

总结:

  • master没挂,则master占有vip且nginx运行在master上
  • master挂了,则slave抢占vip且在slave上运行nginx服务
  • 如果master上的nginx服务挂了,则nginx会自动重启,重启失败后会自动关闭keepalived,这样vip资源也会转移到slave上。
  • master和slave两边都开启nginx服务,无论master还是slave,当其中的一个keepalived服务停止后,vip都会漂移到keepalived服务还在的节点上;
  • 如果要想使nginx服务挂了,vip也漂移到另一个节点,则必须用脚本或者在配置文件里面用shell命令来控制。(nginx服务宕停后会自动启动,启动失败后会强制关闭keepalived,从而致使vip资源漂移到另一台机器上)

主从的缺点在于如果主机一直稳定,那么从机就会一直处于空闲状态,造成了资源的浪费。

主主配置

节点规划

主机名ip软件端口角色
yfm01192.168.199.101nginx、keepalived80主、从节点
yfm02192.168.199.102nginx、keepalived80从、从节点
VIP192.168.199.99、192.168.199.98

192.168.199.101

! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.199.101 #指定smtp服务器地址
   smtp_connect_timeout 30
   router_id LVS_DEVEL1 #负载均衡标识,在局域网内应该是唯一的
}

vrrp_script chk_http_port {
        script "/etc/keepalived/nginx_check.sh"
        interval 2
        weight 2
}

vrrp_instance VI_1 {
    state MASTER #指定该keepalived节点的初始状态
    interface ens33
    virtual_router_id 51 #指定VRRP实例ID
    priority 100 #指定优先级,优先级高的将成为MASTER
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.99 #指定VIP地址
    }
	track_script {
	chk_http_port
   }
}
vrrp_instance VI_2 {
    state BACKUP  #这里修改为BACKUP
    interface ens33
    virtual_router_id 52 #这里修改为52
    priority 90    #修改优先级
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.98  #绑定另外一个VIP
    }

   track_script {
	chk_http_port
   }
}

192.168.199.102

! Configuration File for keepalived
global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.199.101
   smtp_connect_timeout 30
   router_id LVS_DEVEL2
}
vrrp_script chk_http_port {
        script "/etc/keepalived/nginx_check.sh"
        interval 2
        weight 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.99
    }
	track_script {
   chk_http_port
	}
}
vrrp_instance VI_2 {
    state MASTER
    interface ens33
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.98
    }

track_script {
   chk_http_port
}
}

nginx_check.sh

#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
    /usr/sbin/nginx
    sleep 2
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        systemctl stop keepalived
    fi
fi

对VRRP_Instance(VI_1)而言,192.168.199.101为MASTER;对VRRP_Instance(VI_2)而言,192.168.199.102为MASTER。

浏览器访问http://192.168.199.99/,192.168.199.101上的nginx会处理该请求;浏览器访问http://192.168.199.98/,192.168.199.102上的nginx会处理该请求 !

当任何一台服务器上面的keepalived服务停掉后,另外一台服务器上面的keepalived都会进入MASTER状态处理请求。停掉192.168.199.101上的keepalived服务,则192.168.199.102上的keepalived服务的VRRP_Instance(VI_1)进入MASTER状态,并对http://192.168.199.99/和http://192.168.199.98/进行处理!

访问http://192.168.199.99/
在这里插入图片描述

访问http://192.168.199.98/
在这里插入图片描述

lvs + keepalived + nginx 主从模式

节点规划

主机名ip软件端口角色
yfm01192.168.199.101lvs、keepalived80MASTER
yfm02192.168.199.102lvs、keepalived80SLAVE
yfm03192.168.199.103nginx80realserver1
yfm04192.168.199.104nginx80realserver2
VIP192.168.199.99
在yfm01、yfm02节点安装lvs
yum -y install ipvsadm
加载ipvsadm模块进系统
ipvsadm


在yfm03、yfm04节点安装nginx
yum -y install nginx

192.168.199.101、/etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   router_id LVS_Master
}
   
vrrp_instance VI_1 {
    state MASTER               #指定instance初始状态,实际根据优先级决定.backup节点不一样
    interface ens33             #虚拟IP所在网
    virtual_router_id 51       #VRID,相同VRID为一个组,决定多播MAC地址
    priority 100               #优先级,另一台改为90.backup节点不一样
    advert_int 1               #检查间隔
    authentication {
        auth_type PASS         #认证方式,可以是pass或ha
        auth_pass 1111         #认证密码
    }
    virtual_ipaddress {
        192.168.199.99         #VIP
    }
}
   
virtual_server 192.168.199.99 80 {
    delay_loop 6               #服务轮询的时间间隔
    lb_algo wrr                #加权轮询调度,LVS调度算法 rr|wrr|lc|wlc|lblc|sh|sh
    lb_kind DR                 #LVS集群模式 NAT|DR|TUN,其中DR模式要求负载均衡器网卡必须有一块与物理网卡在同一个网段
    #nat_mask 255.255.255.0
    persistence_timeout 50     #会话保持时间
    protocol TCP              #健康检查协议
   
    ## Real Server设置,80就是MySQL连接端口
    real_server 192.168.199.103 80 {
        weight 3  ##权重
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.199.104 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}

192.168.199.102、/etc/keepalived/keepalived.conf

! Configuration File for keepalived
   
global_defs {
   router_id LVS_Backup
}
   
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.99
    }
}
   
virtual_server 192.168.199.99 80 {
    delay_loop 6           
    lb_algo wrr         
    lb_kind DR
   
    persistence_timeout 50  
    protocol TCP          
   
    real_server 192.168.199.103 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.199.104 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}

192.168.199.103、192.168.199.104

新建/etc/init.d/realserver脚本

#!/bin/sh
#chkconfig: 2345 90 10
VIP=192.168.199.99
. /etc/rc.d/init.d/functions
   
case "$1" in
# 禁用本地的ARP请求、绑定本地回环地址
start)
    /sbin/ifconfig lo down
    /sbin/ifconfig lo up
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
    /sbin/sysctl -p >/dev/null 2>&1
    /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up     #在回环地址上绑定VIP,设定掩码,与Direct Server(自身)上的IP保持通信
    /sbin/route add -host $VIP dev lo:0
    echo "LVS-DR real server starts successfully.\n"
    ;;
stop)
    /sbin/ifconfig lo:0 down
    /sbin/route del $VIP >/dev/null 2>&1
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
    ;;
status)
    isLoOn=`/sbin/ifconfig lo:0 | grep "$VIP"`
    isRoOn=`/bin/netstat -rn | grep "$VIP"`
    if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
        echo "LVS-DR real server has run yet."
    else
        echo "LVS-DR real server is running."
    fi
    exit 3
    ;;
*)
    echo "Usage: $0 {start|stop|status}"
    exit 1
esac
exit 0

赋予可执行权限
chomd +x realserver

开机启动
chkconfig realserver on

使用ip addr命令查看yfm01、yfm02,可以看到yfm01上绑定了vip,yfm02没有
在这里插入图片描述

在这里插入图片描述

在yfm01上执行ipvsadm -ln检查连接情况
在这里插入图片描述

修改yfm03、yfm04,nginx欢迎界面,添加上本机ip

whereis nginx
[root@yfm03 nginx]# whereis nginx
nginx: /usr/sbin/nginx /usr/lib64/nginx /etc/nginx /usr/share/nginx /usr/share/man/man3/nginx.3pm.gz /usr/share/man/man8/nginx.8.gz
可以看到nginx配置文件在 /etc/nginx目录

在浏览器访问192.168.199.103、192.168.199.104、192.168.199.99均可访问nginx服务
在这里插入图片描述

总结:

  • 当 MASTER 服务器无法提供服务时,VIP 会在 MASTER 上自动移除,BACKUP 服务器会提升为 MASTER 状态,绑定 VIP 、接管服务。
  • 当 MASTER 修复加入网络后,会自动抢回 VIP ,成为 MASTER 身份。
  • 当后端提供服务nginx服务挂起时,会自动切换至其它nginx服务器。
lvs + keepalived + nginx 主主模式

主主环境相比于主从环境,区别只在于

  1. LVS负载均衡层需要两个VIP。比如192.168.199.98和192.168.199.99
  2. 后端的realserver上要绑定这两个VIP到lo本地回环口上
  3. Keepalived.conf的配置相比于上面的主从模式也有所不同

在yfm03,/etc/init.d目录新建realserver98、realserver99

realserver98

#!/bin/sh
#chkconfig: 2345 90 10
VIP=192.168.199.98
. /etc/rc.d/init.d/functions
   
case "$1" in
 
start)
    /sbin/ifconfig lo down
    /sbin/ifconfig lo up
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
    /sbin/sysctl -p >/dev/null 2>&1
    /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up   
    /sbin/route add -host $VIP dev lo:0
    echo "LVS-DR real server starts successfully.\n"
    ;;
stop)
    /sbin/ifconfig lo:0 down
    /sbin/route del $VIP >/dev/null 2>&1
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
    ;;
status)
    isLoOn=`/sbin/ifconfig lo:0 | grep "$VIP"`
    isRoOn=`/bin/netstat -rn | grep "$VIP"`
    if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
        echo "LVS-DR real server has run yet."
    else
        echo "LVS-DR real server is running."
    fi
    exit 3
    ;;
*)
    echo "Usage: $0 {start|stop|status}"
    exit 1
esac
exit 0

realserver99

#!/bin/sh
#chkconfig: 2345 90 10
VIP=192.168.199.99
. /etc/rc.d/init.d/functions
   
case "$1" in
 
start)
    /sbin/ifconfig lo down
    /sbin/ifconfig lo up
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
    /sbin/sysctl -p >/dev/null 2>&1
    /sbin/ifconfig lo:1 $VIP netmask 255.255.255.255 up   
    /sbin/route add -host $VIP dev lo:1
    echo "LVS-DR real server starts successfully.\n"
    ;;
stop)
    /sbin/ifconfig lo:1 down
    /sbin/route del $VIP >/dev/null 2>&1
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
    ;;
status)
    isLoOn=`/sbin/ifconfig lo:1 | grep "$VIP"`
    isRoOn=`/bin/netstat -rn | grep "$VIP"`
    if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
        echo "LVS-DR real server has run yet."
    else
        echo "LVS-DR real server is running."
    fi
    exit 3
    ;;
*)
    echo "Usage: $0 {start|stop|status}"
    exit 1
esac
exit 0

赋予可执行权限
chomd +x realserver9*

在yfm04执行以下命令,复制刚刚新建的文件
scp root@192.168.199.103:/etc/init.d/realserver9* /etc/init.d/

在yfm03、yfm04执行以下命令
service realserver99 start
service realserver98 start

可以发现两个VIP已经成功绑定到lo上了
在这里插入图片描述

yfm01,/etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   router_id LVS_Master
}
   
vrrp_instance VI_1 {
    state MASTER               #指定instance初始状态,实际根据优先级决定.backup节点不一样
    interface ens33             #虚拟IP所在网
    virtual_router_id 51       #VRID,相同VRID为一个组,决定多播MAC地址
    priority 100               #优先级,另一台改为90.backup节点不一样
    advert_int 1               #检查间隔
    authentication {
        auth_type PASS         #认证方式,可以是pass或ha
        auth_pass 1111         #认证密码
    }
    virtual_ipaddress {
        192.168.199.99         #VIP
    }
}

vrrp_instance VI_2 {
    state BACKUP           
    interface ens33          
    virtual_router_id 52   
    priority 90            
    advert_int 1           
    authentication {
        auth_type PASS      
        auth_pass 1111      
    }
    virtual_ipaddress {
        192.168.199.98
    }
}

virtual_server 192.168.199.99 80 {
    delay_loop 6               #服务轮询的时间间隔
    lb_algo wrr                #加权轮询调度,LVS调度算法 rr|wrr|lc|wlc|lblc|sh|sh
    lb_kind DR                 #LVS集群模式 NAT|DR|TUN,其中DR模式要求负载均衡器网卡必须有一块与物理网卡在同一个网段
    #nat_mask 255.255.255.0
    persistence_timeout 50     #会话保持时间
    protocol TCP              #健康检查协议
   
    ## Real Server设置,80就是MySQL连接端口
    real_server 192.168.199.103 80 {
        weight 3  ##权重
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.199.104 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}


virtual_server 192.168.199.98 80 {
    delay_loop 6               #服务轮询的时间间隔
    lb_algo wrr                #加权轮询调度,LVS调度算法 rr|wrr|lc|wlc|lblc|sh|sh
    lb_kind DR                 #LVS集群模式 NAT|DR|TUN,其中DR模式要求负载均衡器网卡必须有一块与物理网卡在同一个网段
    #nat_mask 255.255.255.0
    persistence_timeout 50     #会话保持时间
    protocol TCP              #健康检查协议
   
    ## Real Server设置,80就是MySQL连接端口
    real_server 192.168.199.103 80 {
        weight 3  ##权重
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.199.104 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}

yfm02,/etc/keepalived/keepalived.conf

! Configuration File for keepalived
   
global_defs {
   router_id LVS_Backup
}
   
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.199.99
    }
}
vrrp_instance VI_2 {
    state Master          
    interface ens33          
    virtual_router_id 52  
    priority 100           
    advert_int 1           
    authentication {
        auth_type PASS      
        auth_pass 1111      
    }
    virtual_ipaddress {
        182.148.15.235    
    }
}


virtual_server 192.168.199.99 80 {
    delay_loop 6           
    lb_algo wrr         
    lb_kind DR
   
    persistence_timeout 50  
    protocol TCP          
   
    real_server 192.168.199.103 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.199.104 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}

virtual_server 192.168.199.98 80 {
    delay_loop 6           
    lb_algo wrr         
    lb_kind DR
   
    persistence_timeout 50  
    protocol TCP          
   
    real_server 192.168.199.103 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.199.104 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}
在yfm01,打开ip_forward路由转发功能
echo "1" > /proc/sys/net/ipv4/ip_forward

在yfm01、yfm02上启动keepalived
service keepalived start

在这里插入图片描述

在这里插入图片描述

可以看到,yfm01已经成功绑定两个VIP,而yfm02没有绑定;

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

四美

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值