nginx和keepalived实现nginx高可用

首先介绍一下Keepalived，它是一个高性能的服务器高可用或热备解决方案，Keepalived主要来防止服务器单点故障的发生问题，可以通过其与Nginx的配合实现web服务端的高可用。Keepalived以VRRP协议为实现基础，用VRRP协议来实现高可用性（HA）.VRRP (Virtual Router Redundancy Protocol)协议是用于实现路由器冗余的协议，VRRP协议将两台或多台路由器设备虚拟成一个设备，对外提供虚拟路由器IP（一个或多个） 我们使用keepalived来管理两台设备的Nginx，并虚拟出一个IP，我们现在两台装有Nginx的设备分别是192.168.118.112和192.168.118.100，那么我们可以虚拟出一个192.168.118.23的IP，外界请求直接访问虚拟IP而不是真正的Nginx，让虚拟IP去访问提供服务的Nginx（注意：高可用是指同一时间提供服务的只有一台设备，提供服务的设备挂掉之后，备份服务器便开始提供服务）

实验环境

服务器	ip地址	vip
keepalived+nginx	192.168.118.112(master)	192.168.118.23
keepalived+nginx	192.168.118.100(backup)	192.168.118.23

（两台虚拟机前面安装服务的配置是一样的）
安装yum 源

[root@localhost ~]# cd /etc/yum.repos.d/
[root@localhost yum.repos.d]# curl -o CentOS7-Base-163.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo
[root@nginx-server yum.repos.d]# sed -i 's/\$releasever/7/g' /etc/yum.repos.d/CentOS7-Base-163.repo
[root@localhost yum.repos.d]# sed -i 's/^enabled=.*/enabled=1/g' /etc/yum.repos.d/CentOS7-Base-163.repo
[root@localhost yum.repos.d]# yum -y install epel-release

关闭防火墙和selinux
不让防火墙开机重启

chkconfig iptables off

[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable firewalld
[root@nginx-server ~]# setenforce 0
[root@localhost ~]# sed -ri 's/(^SELINUX=).*/\1disabled/g' /etc/selinux/config 

下载keepalive
官网
安装nginx和keepalived可参考
详细步骤
下载nginx

yum -y install nginx

部署keepalived
将压缩包放在/usr/src下（两台虚拟机配置相同）

tar -zxvf keepalived-1.2.18.tar.gz -C /usr/local/     #解压
yum install popt-devel  openssl-devel  -y  #下载keepalived 所需要的开发包
cd keepalived-1.2.18/ && ./configure --prefix=/usr/local/keepalived    
make && make install    #编译安装
mkdir /etc/keepalived   #创建目录
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ #复制默认配置文件到默认路径
#复制keepalived 的服务脚本到默认的位置
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ 
ln -s /usr/local/sbin/keepalived /usr/sbin/ 
ln -s /usr/local/keepalived/sbin/keepalived /sbin/

设置服务为开机启动

chkconfig keepalived on
chkconfig nginx on

若keepalived服务没有启动

#解决
[root@localhost sbin]# cd /usr/sbin/
[root@localhost sbin]# rm -f keepalived 
[root@localhost sbin]# cp /usr/local/keepalived/sbin/keepalived  /usr/sbin/

启动服务

systemctl start nginx
systemctl start keepalived

不用脚本 配置文件改动如下 服务
master

[root@localhost keepalived]# pwd
/etc/keepalived

[root@localhost keepalived]# vim keepalived.conf 

! Configuration File for keepalived

global_defs {
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   router_id localhost     #主机名
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33   #本机网卡名
    virtual_router_id 51        #id与backup相同
    priority 100     #优先级
    advert_int 1
    authentication {
        auth_type PASS   #与backup保持一致
        auth_pass 1111   #与backup保持一致
    }
    virtual_ipaddress {
        192.168.118.23/24      #虚拟ip
    }
}

backup

[root@localhost keepalived]# vim keepalived.conf 

! Configuration File for keepalived

global_defs {
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   router_id localhost
}

vrrp_instance VI_1 {
    state BACKUP
    mcast_src_ip 192.168.118.100
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.118.23/24
    }
}

查看虚拟ip(master)

[root@localhost keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:62:03:2c brd ff:ff:ff:ff:ff:ff
    inet 192.168.118.112/24 brd 192.168.118.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.118.23/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::dc5a:a4e4:f673:e7c/64 scope link 
       valid_lft forever preferred_lft forever

关闭master服务，查看backup虚拟ip，实现地址漂移

[root@localhost keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:df:b0:34 brd ff:ff:ff:ff:ff:ff
    inet 192.168.118.100/24 brd 192.168.118.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.118.23/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::2a59:9877:821d:98f6/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
    link/ether 52:54:00:99:dd:d4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
    link/ether 52:54:00:99:dd:d4 brd ff:ff:ff:ff:ff:ff

访问虚拟ip

编辑nginx配置文件，以方便验证（master）

[root@localhost html]# pwd
/usr/share/nginx/html
[root@localhost html]# mv index.html jiade.html
[root@localhost html]# echo "192.168.118.112" > index.html
[root@localhost html]# ls
404.html  index.html  nginx-logo.png
50x.html  jiade.html  poweredby.png
[root@localhost html]# cat index.html 
192.168.118.112

验证：

backup

[root@localhost home]# cd /usr/share/nginx/html/
[root@localhost html]# ls
404.html  50x.html  index.html  nginx-logo.png  poweredby.png
[root@localhost html]# mv index.html jiade.html
[root@localhost html]# echo "192.168.118.100" > index.html

验证

添加启动脚本

[root@localhost keepalived]# ls
keepalived.conf  nginx_check.sh
[root@localhost keepalived]# vim nginx_check.sh 

#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
        systemctl restart nginx
        sleep 2
        if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
                killall keepalived
        fi
fi
#给脚本权限
[root@localhost keepalived]# chmod 777 nginx_check.sh 
[root@localhost keepalived]# ll
总用量 8
-rw-r--r--. 1 root root 3559 10月 29 11:14 keepalived.conf
-rwxrwxrwx. 1 root root  182 10月 29 03:11 nginx_check.sh

脚本详解：

• 检查当前nginx的进程数量并把查询到的进程数量赋值给变量A。
• 如果查询到的nginx的进程数量是0的话，就执行if条件里的内容
• 若检查到当前没有nginx进程，就尝试去启动nginx
• 启动nginx之后休眠2秒
• 如果nginx的进程数还是0的话，就认为nginx已经挂掉了，需要杀掉这个节点上所有的keepalived进程。

配置文件调用脚本

（master）

[root@localhost keepalived]# vim keepalived.conf 

! Configuration File for keepalived

global_defs {
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   router_id localhost
}
#添加这段内容
#每隔2秒中去执行/etc/keepalived/nginx_check.sh脚本一次，这项检查从开始便一直进行，interval表示间隔时间，weight -20的意思是，脚本执行成功后把192.168.156.11这个节点的优先级降低20
vrrp_script chk_nginx {
   script "/etc/keepalived/nginx_check.sh"
   interval 2
   weight -20
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    #增加这段内容
    #调用脚本"chk_nginx"与我们在上面定义的定时执行脚本配置（vrrp_script chk_nginx）的名称要一样。 
    track_script {
       chk_nginx
    }
    virtual_ipaddress {
        192.168.118.23/24
    }
}

virtual_server 192.168.200.100 443 {
    delay_loop 6

backup

[root@localhost keepalived]# vim keepalived.conf 

! Configuration File for keepalived

global_defs {
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   router_id localhost
}

vrrp_ script chk_nginx {
   script "/etc/keepalived/nginx_check.sh"
   interval 2
   weight -20
}
vrrp_instance VI_1 {
    state BACKUP
    mcast_src_ip 192.168.118.100
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
       chk_nginx
    }
    virtual_ipaddress {
        192.168.118.23/24
    }

验证：
访问虚拟ip出现的是master默认页内容

关闭master的keepalived服务，访问虚拟ip

[root@localhost keepalived]# service keepalived stop 
Stopping keepalived (via systemctl):                       [  确定  ]

等我们的master设备故障修复之后，我们重启keepalived

[root@localhost keepalived]# service keepalived start
Starting keepalived (via systemctl):                       [  确定  ]

再次验证

由于我们把keepalived配置成了服务，并且设置成了开机自启动，下面我们把两台设备都重启，重启的命令是reboot。
重启后，我们使用命令service keepalived status查看keepalived是否自己启动了，而且我们在nginx_check.sh脚本中会自动开启nginx。
重启虚拟机，发现服务都重启了

[root@localhost ~]# ss -antl
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128       *:111                   *:*                  
LISTEN     0      128       *:80                    *:*                  
LISTEN     0      5      192.168.122.1:53                    *:*                  
LISTEN     0      128       *:22                    *:*                  
LISTEN     0      128    127.0.0.1:631                   *:*                  
LISTEN     0      100    127.0.0.1:25                    *:*                  
LISTEN     0      128      :::111                  :::*                  
LISTEN     0      128      :::80                   :::*                  
LISTEN     0      128      :::22                   :::*                  
LISTEN     0      128     ::1:631                  :::*                  
LISTEN     0      100     ::1:25                   :::*                  
[root@localhost ~]# systemctl status keepalived.service
● keepalived.service - SYSV: Start and stop Keepalived
  Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
  Active: active (running) since 一 2018-10-29 11:49:53 UTC; 2min 52s ago
    Docs: man:systemd-sysv-generator(8)
 Process: 1158 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
Main PID: 1176 (keepalived)
  CGroup: /system.slice/keepalived.service
          ├─1176 keepalived -D
          ├─1180 keepalived -D
          └─1181 keepalived -D

下面我还可以人为让nginx的配置文件出错，这样nginx_check.sh脚本文件在去尝试启动nginx时发现启动不成功，便会将keepalived也都杀掉。
人为改动master的Nginx配置文件

[root@localhost nginx]# pwd
/etc/nginx

[root@localhost nginx]# vim nginx.conf

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/



#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
#将下面三行注释
#events {
#    worker_connections 1024;
#}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

nginx服务不能正常启动

[root@localhost nginx]# systemctl restart nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.

重启master虚拟机，查看nginx和keepalived状态

[root@localhost ~]# ss -antl
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128       *:22                    *:*                  
LISTEN     0      100    127.0.0.1:25                    *:*                  
LISTEN     0      128      :::22                   :::*                  
LISTEN     0      100     ::1:25                   :::*                  
[root@localhost ~]# service keepalived status
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: active (running) since 一 2018-10-29 20:01:48 CST; 1min 38s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1079 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
 Main PID: 1102 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─1102 keepalived -D
           ├─1105 keepalived -D
           ├─1106 keepalived -D
           ├─2090 keepalived -D
           ├─2091 /bin/bash /etc/keepalived/nginx_check.sh
           └─2103 sleep 2

10月 29 20:03:06 localhost.localdomain Keepalived_vrrp[1106]: Process ...
10月 29 20:03:08 localhost.localdomain Keepalived_vrrp[1106]: Process ...
10月 29 20:03:10 localhost.localdomai

可以看到，nginx服务没有启动，但是keepalived服务仍在运行，这就有问题了，因为我们的脚本执行的是若nginx进程数为0 ，就杀死所有的keepalived进程，所以，正常情况下，keepalived服务不应该开启，我们尝试在外面运行脚本，找出问题

[root@localhost ~]# cd /etc/keepalived/
[root@localhost keepalived]# sh -x nginx_check.sh 
++ wc -l
++ ps -C nginx --no-header
+ A=0
+ '[' 0 -eq 0 ']'
+ systemctl restart nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
+ sleep 2
++ wc -l
++ ps -C nginx --no-header
+ '[' 0 -eq 0 ']'
+ killall keepalived
nginx_check.sh:行7: killall: 未找到命令

发现没有killall命令，安装killall

[root@localhost keepalived]# yum -y install psmisc-22.20-15.el7.x86_64li

再次查看keepalived状态

[root@localhost keepalived]# service keepalived status
● keepalived.service - SYSV: Start and stop Keepalived
   Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
   Active: inactive (dead) since 一 2018-10-29 20:14:21 CST; 32s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 7840 ExecStop=/etc/rc.d/init.d/keepalived stop (code=exited, status=0/SUCCESS)
  Process: 1079 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)

这个时候我们访问虚拟ip，就只能访问backup提供的nginx服务了
至此，我们使用keepalived搭建nginx高可用便搭建完了。需要说明的是，keepalived不仅可以用于nginx的高可用，还可以用于redis、mysql等等所有服务的高可用。