Centos7下搭建Nginx Keepalived Cluster

操作环境

CentOS Linux release 7.4.1708 (Core) 

nginx version: nginx/1.12.2

Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2 

Nginx Master Node：10.10.200.3

Nginx Backup Node：10.10.200.2

VIP:10.10.200.100

---

操作步骤

安装nginx

因为主要讲述keepalived配置步骤，nginx就用最简单模式安装

#yum -y install nginx

为了方面测试分别修改2台nginx server中的index.html文件如下：

Nginx Server：10.10.200.3

<h1>Nginx</h1>
<h1>10.10.200.3</h1>

 Nginx Server：10.10.200.2

<h1>Nginx</h1>
<h1>10.10.200.2</h1>

这样我们可以通过curl获取信息如下：

[root@nginx-backup sbin]# curl -l http://10.10.200.3
<h1>Nginx</h1>
<h1>10.10.200.3</h1>
[root@nginx-backup sbin]# curl -l http://10.10.200.2
<h1>Nginx</h1>
<h1>10.10.200.2</h1>

安装配置Keepalived

#yum -y install ipset keepalived

注意一定要安装ipset，否则后面再keepalived的过程中，会报错！！！

在Nginx Master&Backup Node上编辑nginx监控脚本，我们将脚本放在/usr/local/keepalived/sbin目录下

[root@nginx-backup sbin]# vi check_nginx_alive.sh 
#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
    /usr/local/nginx/sbin/nginx
    sleep 2
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        systemctl stop keepalived.service
                exit 1
    fi
fi
exit 0

添加可执行权限

# chmod a+x check_nginx_alive.sh

Nginx Master Node（10.10.200.3）配置文件如下：

! Configuration File for keepalived
       vrrp_script check_nginx_alive {
           script "/usr/local/keepalived/sbin/check_nginx_alive.sh"
           interval 2
           weight -10
       }
       
       global_defs {
          notification_email {
           administrator@qq.com
         }
         script_user root  //记得添加
         enable_script_security  //记得添加
         notification_email_from Alexandre.Cassen@firewall.loc
         smtp_server smtp.exmail.qq.com
         smtp_connect_timeout 30
         router_id LVS_DEVEL_1  //唯一ID，要与Backup Node不同
         vrrp_skip_check_adv_addr
         vrrp_strict
         vrrp_garp_interval 0
         vrrp_gna_interval 0
      }
      
      vrrp_instance VI_1 { 
          state MASTER  //注意在Master中该参数为Master，在Backup中该参数为Backup
          interface em2   //VIP使用的网络端口
          virtual_router_id 51   //该数值要与BackUp Node相同
          priority 102    //Master Node的数值要大于Backup Node 
          advert_int 1
          authentication {   //Master Node与Backup Node该项设置要相同
              auth_type PASS
              auth_pass 1111
          }
          virtual_ipaddress {
              10.10.200.100/24    //VIP Address
          }
          track_script {
              check_nginx_alive
          }
      }

Nginx Backup Node（10.10.200.2）配置文件如下：

! Configuration File for keepalived
vrrp_script check_nginx_alive {
    script "/usr/local/keepalived/sbin/check_nginx_alive.sh"
    interval 2
    weight -10
}

global_defs {
   notification_email {
        administrator@qq.com     
   }
   script_user root
   enable_script_security
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server smtp.exmail.qq.com
   smtp_connect_timeout 30
   router_id LVS_DEVEL_2
   vrrp_skip_check_adv_addr
   script_user root
   enable_script_security
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {
    state BACKUP
    interface em2
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.10.200.100/24
    }
    track_script {
        check_nginx_alive
    }
}

配置完成后，分别在2台Nginx Server上启动keepalived和Nginx。

在Master Node上通过 ip addr 可以查看到VIP：

3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 84:8f:69:da:60:7b brd ff:ff:ff:ff:ff:ff
    inet 10.10.200.3/24 brd 10.10.200.255 scope global em2
       valid_lft forever preferred_lft forever
    inet 10.10.200.100/24 scope global secondary em2
       valid_lft forever preferred_lft forever
    inet6 fe80::f1d5:a5ab:3d0d:ef6/64 scope link 
       valid_lft forever preferred_lft forever

也可以ping通VIP：

[root@kvm-server ~]# ping 10.10.200.100
PING 10.10.200.100 (10.10.200.100) 56(84) bytes of data.
64 bytes from 10.10.200.100: icmp_seq=1 ttl=64 time=0.052 ms
64 bytes from 10.10.200.100: icmp_seq=2 ttl=64 time=0.040 ms
64 bytes from 10.10.200.100: icmp_seq=3 ttl=64 time=0.040 ms
64 bytes from 10.10.200.100: icmp_seq=4 ttl=64 time=0.049 ms

通过curl 查看10.10.200.100的页面，此时通过10.10.200.100访问的数据时Nginx Master Node（10.10.200.3）的页面数据。

[root@kvm-server ~]# curl -l http://10.10.200.100
<h1>Nginx</h1>
<h1>10.10.200.3</h1>

下面测试下，停掉Nginx Master Node的nginx服务，keepalived是否会将nginx服务切换至Nginx Backup Node（10.10.200.2）上面。

#/etc/init.d/nginx stop

此时查看Nginx BackUp Node（10.10.200.2）上面的日志信息，VIP已经切换过来

#tail -f /var/log/message
Jul 30 15:04:55 nginx-backup Keepalived_vrrp[23293]: VRRP_Instance(VI_1) forcing a new MASTER election
Jul 30 15:04:56 nginx-backup Keepalived_vrrp[23293]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: VRRP_Instance(VI_1) Entering MASTER STATE
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: VRRP_Instance(VI_1) setting protocol iptable drop rule
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: VRRP_Instance(VI_1) setting protocol VIPs.
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on em2 for 10.10.200.100
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:04:57 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:05:02 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:05:02 nginx-backup Keepalived_vrrp[23293]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on em2 for 10.10.200.100
Jul 30 15:05:02 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:05:02 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:05:02 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100
Jul 30 15:05:02 nginx-backup Keepalived_vrrp[23293]: Sending gratuitous ARP on em2 for 10.10.200.100

通过ip addr查看

3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 84:8f:69:da:5e:68 brd ff:ff:ff:ff:ff:ff
    inet 10.10.200.2/24 brd 10.10.200.255 scope global em2
       valid_lft forever preferred_lft forever
    inet 10.10.200.100/24 scope global secondary em2
       valid_lft forever preferred_lft forever
    inet6 fe80::d7ca:173b:3e56:65f5/64 scope link 
       valid_lft forever preferred_lft forever

通过 curl 获取10.10.200.100页面数据，此时的页面数据应该为Nginx BackUp Node（10.10.200.2）的数据

[root@nginx-backup sbin]# curl -l http://10.10.200.100
<h1>Nginx</h1>
<h1>10.10.200.2</h1>

经过上面的操作，说明keepalived能正常对nginx cluster的master&backup node进行切换。