首先介绍一下Keepalived,它是一个高性能的服务器高可用或热备解决方案,Keepalived主要来防止服务器单点故障的发生问题,可以通过其与Nginx的配合实现web服务端的高可用。Keepalived以VRRP协议为实现基础,用VRRP协议来实现高可用性(HA).VRRP (Virtual Router Redundancy Protocol)协议是用于实现路由器冗余的协议,VRRP协议将两台或多台路由器设备虚拟成一个设备,对外提供虚拟路由器IP(一个或多个) 我们使用keepalived来管理两台设备的Nginx,并虚拟出一个IP,我们现在两台装有Nginx的设备分别是192.168.118.112和192.168.118.100,那么我们可以虚拟出一个192.168.118.23的IP,外界请求直接访问虚拟IP而不是真正的Nginx,让虚拟IP去访问提供服务的Nginx(注意:高可用是指同一时间提供服务的只有一台设备,提供服务的设备挂掉之后,备份服务器便开始提供服务)
实验环境
服务器 | ip地址 | vip |
---|---|---|
keepalived+nginx | 192.168.118.112(master) | 192.168.118.23 |
keepalived+nginx | 192.168.118.100(backup) | 192.168.118.23 |
(两台虚拟机前面安装服务的配置是一样的)
安装yum 源
[root@localhost ~]# cd /etc/yum.repos.d/
[root@localhost yum.repos.d]# curl -o CentOS7-Base-163.repo http://mirrors.163.com/.help/CentOS7-Base-163.repo
[root@nginx-server yum.repos.d]# sed -i 's/\$releasever/7/g' /etc/yum.repos.d/CentOS7-Base-163.repo
[root@localhost yum.repos.d]# sed -i 's/^enabled=.*/enabled=1/g' /etc/yum.repos.d/CentOS7-Base-163.repo
[root@localhost yum.repos.d]# yum -y install epel-release
关闭防火墙和selinux
不让防火墙开机重启
chkconfig iptables off
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable firewalld
[root@nginx-server ~]# setenforce 0
[root@localhost ~]# sed -ri 's/(^SELINUX=).*/\1disabled/g' /etc/selinux/config
下载keepalive
官网
安装nginx和keepalived可参考
详细步骤
下载nginx
yum -y install nginx
部署keepalived
将压缩包放在/usr/src下(两台虚拟机配置相同)
tar -zxvf keepalived-1.2.18.tar.gz -C /usr/local/ #解压
yum install popt-devel openssl-devel -y #下载keepalived 所需要的开发包
cd keepalived-1.2.18/ && ./configure --prefix=/usr/local/keepalived
make && make install #编译安装
mkdir /etc/keepalived #创建目录
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ #复制默认配置文件到默认路径
#复制keepalived 的服务脚本到默认的位置
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
ln -s /usr/local/sbin/keepalived /usr/sbin/
ln -s /usr/local/keepalived/sbin/keepalived /sbin/
设置服务为开机启动
chkconfig keepalived on
chkconfig nginx on
若keepalived服务没有启动
#解决
[root@localhost sbin]# cd /usr/sbin/
[root@localhost sbin]# rm -f keepalived
[root@localhost sbin]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
启动服务
systemctl start nginx
systemctl start keepalived
不用脚本 配置文件改动如下 服务
master
[root@localhost keepalived]# pwd
/etc/keepalived
[root@localhost keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
router_id localhost #主机名
}
vrrp_instance VI_1 {
state MASTER
interface ens33 #本机网卡名
virtual_router_id 51 #id与backup相同
priority 100 #优先级
advert_int 1
authentication {
auth_type PASS #与backup保持一致
auth_pass 1111 #与backup保持一致
}
virtual_ipaddress {
192.168.118.23/24 #虚拟ip
}
}
backup
[root@localhost keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
router_id localhost
}
vrrp_instance VI_1 {
state BACKUP
mcast_src_ip 192.168.118.100
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.118.23/24
}
}
查看虚拟ip(master)
[root@localhost keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:62:03:2c brd ff:ff:ff:ff:ff:ff
inet 192.168.118.112/24 brd 192.168.118.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.118.23/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::dc5a:a4e4:f673:e7c/64 scope link
valid_lft forever preferred_lft forever
关闭master服务,查看backup虚拟ip,实现地址漂移
[root@localhost keepalived]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:df:b0:34 brd ff:ff:ff:ff:ff:ff
inet 192.168.118.100/24 brd 192.168.118.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.118.23/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::2a59:9877:821d:98f6/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:99:dd:d4 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:99:dd:d4 brd ff:ff:ff:ff:ff:ff
访问虚拟ip
编辑nginx配置文件,以方便验证(master)
[root@localhost html]# pwd
/usr/share/nginx/html
[root@localhost html]# mv index.html jiade.html
[root@localhost html]# echo "192.168.118.112" > index.html
[root@localhost html]# ls
404.html index.html nginx-logo.png
50x.html jiade.html poweredby.png
[root@localhost html]# cat index.html
192.168.118.112
验证:
backup
[root@localhost home]# cd /usr/share/nginx/html/
[root@localhost html]# ls
404.html 50x.html index.html nginx-logo.png poweredby.png
[root@localhost html]# mv index.html jiade.html
[root@localhost html]# echo "192.168.118.100" > index.html
验证
添加启动脚本
[root@localhost keepalived]# ls
keepalived.conf nginx_check.sh
[root@localhost keepalived]# vim nginx_check.sh
#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];then
systemctl restart nginx
sleep 2
if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
killall keepalived
fi
fi
#给脚本权限
[root@localhost keepalived]# chmod 777 nginx_check.sh
[root@localhost keepalived]# ll
总用量 8
-rw-r--r--. 1 root root 3559 10月 29 11:14 keepalived.conf
-rwxrwxrwx. 1 root root 182 10月 29 03:11 nginx_check.sh
脚本详解:
- 检查当前nginx的进程数量并把查询到的进程数量赋值给变量A。
- 如果查询到的nginx的进程数量是0的话,就执行if条件里的内容
- 若检查到当前没有nginx进程,就尝试去启动nginx
- 启动nginx之后休眠2秒
- 如果nginx的进程数还是0的话,就认为nginx已经挂掉了,需要杀掉这个节点上所有的keepalived进程。
配置文件调用脚本
(master)
[root@localhost keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
router_id localhost
}
#添加这段内容
#每隔2秒中去执行/etc/keepalived/nginx_check.sh脚本一次,这项检查从开始便一直进行,interval表示间隔时间,weight -20的意思是,脚本执行成功后把192.168.156.11这个节点的优先级降低20
vrrp_script chk_nginx {
script "/etc/keepalived/nginx_check.sh"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
#增加这段内容
#调用脚本"chk_nginx"与我们在上面定义的定时执行脚本配置(vrrp_script chk_nginx)的名称要一样。
track_script {
chk_nginx
}
virtual_ipaddress {
192.168.118.23/24
}
}
virtual_server 192.168.200.100 443 {
delay_loop 6
backup
[root@localhost keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
router_id localhost
}
vrrp_ script chk_nginx {
script "/etc/keepalived/nginx_check.sh"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
mcast_src_ip 192.168.118.100
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_nginx
}
virtual_ipaddress {
192.168.118.23/24
}
验证:
访问虚拟ip出现的是master默认页内容
关闭master的keepalived服务,访问虚拟ip
[root@localhost keepalived]# service keepalived stop
Stopping keepalived (via systemctl): [ 确定 ]
等我们的master设备故障修复之后,我们重启keepalived
[root@localhost keepalived]# service keepalived start
Starting keepalived (via systemctl): [ 确定 ]
再次验证
由于我们把keepalived配置成了服务,并且设置成了开机自启动,下面我们把两台设备都重启,重启的命令是reboot。
重启后,我们使用命令service keepalived status查看keepalived是否自己启动了,而且我们在nginx_check.sh脚本中会自动开启nginx。
重启虚拟机,发现服务都重启了
[root@localhost ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:111 *:*
LISTEN 0 128 *:80 *:*
LISTEN 0 5 192.168.122.1:53 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 128 127.0.0.1:631 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::111 :::*
LISTEN 0 128 :::80 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 ::1:631 :::*
LISTEN 0 100 ::1:25 :::*
[root@localhost ~]# systemctl status keepalived.service
● keepalived.service - SYSV: Start and stop Keepalived
Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
Active: active (running) since 一 2018-10-29 11:49:53 UTC; 2min 52s ago
Docs: man:systemd-sysv-generator(8)
Process: 1158 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
Main PID: 1176 (keepalived)
CGroup: /system.slice/keepalived.service
├─1176 keepalived -D
├─1180 keepalived -D
└─1181 keepalived -D
下面我还可以人为让nginx的配置文件出错,这样nginx_check.sh脚本文件在去尝试启动nginx时发现启动不成功,便会将keepalived也都杀掉。
人为改动master的Nginx配置文件
[root@localhost nginx]# pwd
/etc/nginx
[root@localhost nginx]# vim nginx.conf
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
#将下面三行注释
#events {
# worker_connections 1024;
#}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
nginx服务不能正常启动
[root@localhost nginx]# systemctl restart nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
重启master虚拟机,查看nginx和keepalived状态
[root@localhost ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
[root@localhost ~]# service keepalived status
● keepalived.service - SYSV: Start and stop Keepalived
Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
Active: active (running) since 一 2018-10-29 20:01:48 CST; 1min 38s ago
Docs: man:systemd-sysv-generator(8)
Process: 1079 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
Main PID: 1102 (keepalived)
CGroup: /system.slice/keepalived.service
├─1102 keepalived -D
├─1105 keepalived -D
├─1106 keepalived -D
├─2090 keepalived -D
├─2091 /bin/bash /etc/keepalived/nginx_check.sh
└─2103 sleep 2
10月 29 20:03:06 localhost.localdomain Keepalived_vrrp[1106]: Process ...
10月 29 20:03:08 localhost.localdomain Keepalived_vrrp[1106]: Process ...
10月 29 20:03:10 localhost.localdomai
可以看到,nginx服务没有启动,但是keepalived服务仍在运行,这就有问题了,因为我们的脚本执行的是若nginx进程数为0 ,就杀死所有的keepalived进程,所以,正常情况下,keepalived服务不应该开启,我们尝试在外面运行脚本,找出问题
[root@localhost ~]# cd /etc/keepalived/
[root@localhost keepalived]# sh -x nginx_check.sh
++ wc -l
++ ps -C nginx --no-header
+ A=0
+ '[' 0 -eq 0 ']'
+ systemctl restart nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
+ sleep 2
++ wc -l
++ ps -C nginx --no-header
+ '[' 0 -eq 0 ']'
+ killall keepalived
nginx_check.sh:行7: killall: 未找到命令
发现没有killall命令,安装killall
[root@localhost keepalived]# yum -y install psmisc-22.20-15.el7.x86_64li
再次查看keepalived状态
[root@localhost keepalived]# service keepalived status
● keepalived.service - SYSV: Start and stop Keepalived
Loaded: loaded (/etc/rc.d/init.d/keepalived; bad; vendor preset: disabled)
Active: inactive (dead) since 一 2018-10-29 20:14:21 CST; 32s ago
Docs: man:systemd-sysv-generator(8)
Process: 7840 ExecStop=/etc/rc.d/init.d/keepalived stop (code=exited, status=0/SUCCESS)
Process: 1079 ExecStart=/etc/rc.d/init.d/keepalived start (code=exited, status=0/SUCCESS)
这个时候我们访问虚拟ip,就只能访问backup提供的nginx服务了
至此,我们使用keepalived搭建nginx高可用便搭建完了。需要说明的是,keepalived不仅可以用于nginx的高可用,还可以用于redis、mysql等等所有服务的高可用。