Nginx+Keepalived搭建高可用负载均衡集群

 

一。 环境说明

前端双Nginx+keepalived,nginx反向代理到后端的tomcat集群实现负载均衡，Keepalived实现集群高可用.

  • 操作系统： Centos 6.6_X64

  • Nginx版本： nginx-1.9.5

  • Keepalived版本：keepalived-1.2.13  


结构:

  Keepalived+nginx-MASTER：10.6.1.210
         Keepalived+nginx-BACKUP:  10.6.1.211
         VIP: 10.6.1.1.214
         Realsever:10.6.1.212,10.6.1.213  


wKiom1YXaVexlDmPAAGc7L6tONs011.jpg



二、安装

    前端两台主机分别安装nginx和keepalived。

1.    安装keepalived

Yum install keepalived –y

默认安装目录: /etc/keepalived/

配置文件:/etc/keepalived/keepalived.conf

 

2.    安装Nginx

1) 安装nginx所依赖的包.


yum install gcc gcc-c++ autoconf automake zlib zlib-developenssl openssl-devel pcre pcre-deve –y


2）Nginx官网下载安装包 (http://nginx.org/en/download.html)

此处版本选择nginx-1.9.5.tar.gz

 

3）编译安装

tar –zxvf nginx-1.9.5.tar.gz

cd nginx-1.9.5

./configure

Make && make install

完成

 

三、配置

1）前端两台主机nginx的配置完全一样

Vim /usr/local/nginx/conf/nginx.conf

upstream tomcat_8080 {

        server10.6.1.212:8080;

        server10.6.1.213:8080;

        ip_hash;

        }

 

 

    server {

        listen       80;

       server_name  test;

 

        #charsetkoi8-r;

 

       #access_log logs/host.access.log  main;

 

        location /{

        proxy_passhttp://tomcat_8080;

 

        }

 

 



2）nginx_master的keepalived配置

Vim /etc/keepalived/keepalived.conf

global_defs {

   notification_email {

     sxl_youcun@qq.com

   }

   notification_email_from sxl_youcun@qq.com

   smtp_server 127.0.0.1

#   smtp_connect_timeout 30

   router_id LVS_DEVEL

}

 

vrrp_scriptcheck_nginx {

 #检查nginx状态的脚本,后面会介绍.同理mysql也可以这样做。

    script"/etc/keepalived/check_nginx.sh"

    # 执行间隔2秒

    interval 2

}

 

#VIP1

vrrp_instance VI_1{

    state BACKUP      #两台主机都是BACKUP

    interface eth0

#同一keepalived集群的virtual_router_id必须相同，默认51   

virtual_router_id 51

    priority 100  #主的优先级高

    advert_int 1

#不抢占：如果集群里已存在MASTER状态的主机，即使优先级高于MASTER也不抢占为MASTER。只在优先级高的主机上设置即可。

    nopreempt

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

#虚拟IP

        10.6.1.214

    }

    track_script {

        check_nginx

    }

}



3）nginx_backup的keepalived配置

global_defs {

   notification_email {

     sxl_youcun@qq.com

   }

   notification_email_from sxl_youcun@qq.com

   smtp_server 127.0.0.1

#   smtp_connect_timeout 30

   router_id LVS_DEVEL

}

 

vrrp_scriptcheck_nginx {

    script"/etc/keepalived/check_nginx.sh"

    # 执行间隔2秒

    interval 2

}

 

#VIP1

vrrp_instance VI_1{

    state BACKUP

    interface eth0

    virtual_router_id 51

    priority 80

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.6.1.214

    }

    track_script {

        check_nginx

    }

}



4）防火墙设置

Iptables -I INPUT -d 224.0.0.18 -j ACCEPT

或者直接关闭防火墙service iptables stop



    VRRP报文是通过IP多播形式发送的，组播地址224.0.0.18是VRRP报文的目的地址。
    两个主机都是BACKUP，如果同时启动keepalived，VRRP协议通过竞选使优先级高的主机做为MASTER。如果防火墙没有允许VRRP报文通过的话，两个BACKUP都会成为MASTER，你会发现两个主机都启动了虚拟IP。
    5）nginx状态检查脚本check_nginx.sh

 Vim /etc/keepalived/check_nginx.sh

#!/bin/bash

#This script is used to check_nginx status for keepalived

KK=`ps -ef |grep "nginx: master process" | grep -v grep | wc -l`

if [ $KK -eq 0 ]

   then

echo "`date`nginx is dead,prepare to startnginx">>/usr/local/nginx/logs/nginx_status.log 2>&1

  /usr/local/nginx/sbin/nginx_start.sh

   sleep 5

   else

   echo "`date` nginx isOK">>/usr/local/nginx/logs/nginx_status.log 2>&1

fi



    6）开启keepalived的日志

    编辑/etc/sysconfig/keepalived：

1

KEEPALIVED_OPTIONS="-D -d -S 0"

编辑/etc/rsyslog.conf：

# 配置文件最后面加上下面一行

local0.*                /var/log/keepalived.log

 

重启rsyslog：

service rsyslog restart

按上面配置后，keepalived会把日志记录到/var/log/keepalived.log。

 

7）启动服务

设置nginx环境变量。

+++++++++++++++++++++++++++++++++++++++++++++

NGINX=/usr/local/nginx

PATH=$PATH:$HOME/bin:$NGINX/sbin

export NGINX PATH

++++++++++++++++++++++++++++++++++++++++++++++

 

启动nginx, 用脚本启动, 脚本如下：

+++++++++++++++++++++++++++++++++++++++++++

#!/bin/bash

. $HOME/.bash_profile

nohup /usr/local/nginx/sbin/nginx -c/usr/local/nginx/conf/nginx.conf >/dev/null 2>&1 &

+++++++++++++++++++++++++++++++++++++++++++++++++

启动keepalived

service keepalived start

 

    四、验证

1. nginx_master和nginx_slave同时启动keepalived，观察日志/var/log/keepalived.log，你会发现nginx_master抢占为MASTER，绑定了虚拟IP。

    nginx_master：

wKioL1YXaXCgI9lbAAHsPyWvqjk146.jpg

 

    nginx_backup：

wKiom1YXaVeTgMBBAAHBk9q8lfI272.jpg

    

2. 把nginx_master的keepalived服务停掉或者重启系统，同时不断的ping虚拟IP。会出现短暂的中断或者时延较大，虚拟IP会漂移到nginx_backup上面：

wKioL1YXaXCxy77QAAOP8_2_ROc480.jpg

    nginx_backup：


wKiom1YXaaKgkyOmAAIv4SlmKVI118.jpg


注意观察在主备切换时nginx_backup日志keepalived.log的变化，如下：

VRRP_Instance(VI_1) Transition to MASTER STATE

VRRP_Instance(VI_1) Entering MASTER STATE

VRRP_Instance(VI_1) setting protocol VIPs.

VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.6.1.214

Netlink reflector reports IP 10.6.1.214 added

VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.6.1.214   

 

3. 最后把nginx_master的keepalived服务开启，虚拟IP并没有漂移回到nginx_master，这是因为nginx_master开启了不抢占模式，即使优先级高，也不会抢占MASTER。