一:Nginx高可用集群解决的问题?
在我们负载均衡配置中有一台非常关键的服务器,nginx反向代理服务器,所有的任务分发都是从他这里发出的, 如果这台反向代理服务器挂了,整个流程全部down了,所以需要对负载均衡进行监听,添加一台备份服务器做容灾。
二:Nginx高可用集群实现思路
1:如上所述我们需要有个程序一直去监听nginx是否down,这里我们使用Keepalived来做
2:当我们发现nginx反向代理服务器down了,我们需要找一台服务器做替代
当nginx主服务故障,使用备份服务器的时候,对外暴露的IP会发生改变,为了减少web端的操作,我们需要统一对外的IP,这就是所谓的虚拟IP, 虚拟IP解决的问题就是当我的nginx故障切换了,对外暴露的IP始终是不变的, 不会对web层有影响。
主要是采用 nginx负载均衡 + 监控(keepalived)来实现,keepalived来监听Nginx是否存活,同时维护 虚拟IP对外访问解决静态路由的单点故障。
三:Keepalived+Nginx高可用集群具体配置
具体架构如下:
192.168.8.203:nginx(主) Vip(虚拟IP): 192.168.8.222
192.168.8.200:nginx(备)
192.168.8.201:server1
192.168.8.202:server2
3.1 :主备服务器配置负载均衡
主服务器:192.168.8.203
vim /usr/local/nginx/conf/nginx.conf
http {
....
upstream myServer {
ip_hash;
server 192.168.8.201:8080 weight=1 max_fails=1 fail_timeout=10;
server 192.168.8.202:8080 weight=1 max_fails=1 fail_timeout=10;
}
log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"' '$connection $upstream_addr ' 'upstream_response_time $upstream_response_time request_time $request_time ';
access_log /var/log/nginx/access.log main;
....
}
在单个server块添加反向代理
vim /usr/local/nginx/conf.d/test.conf
#access_log /var/log/nginx/test/access.log;
....
location / { # 请求全部跳转
proxy_pass http://myServer;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
....
}
备份slave服务器配置步骤和主服务器一样
ps: 配置完记得重启nginx
3.2 :主备服务器安装配置keepalived
1:安装keepalived
yum install keepalived –y
2:配置主服务器keepalived
vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
iwester@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
script_user root
enable_script_security
router_id LVS_01
}
vrrp_script chk_nginx_port {
script "/usr/local/nginx/src/nginx_check.sh"
interval 5
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
unicast_src_ip 192.168.8.203
unicast_peer {
192.168.8.203
192.168.8.200
}
virtual_ipaddress {
192.168.8.222/24
}
track_script
{
chk_nginx_port
}
}
3:配置备份服务器keepalived
vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
iwester@163.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
script_user root
enable_script_security
router_id LVS_02
}
vrrp_script chk_nginx_port {
script "/usr/local/nginx/src/nginx_check.sh"
interval 5
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
unicast_src_ip 192.168.8.200
unicast_peer {
192.168.8.203
192.168.8.200
}
virtual_ipaddress {
192.168.8.222/24
}
track_script
{
chk_nginx_port
}
}
4:主备库都要创建nginx的监控脚本, 如果不适用keepalived自带的vrrp_script也可以自己写一个脚本,常驻后台去判断nginx是否存活!
mkdir -p /usr/local/nginx/src
vim /usr/local/nginx/src/nginx_check.sh
#!/bin/sh
nginxpid=$(ps -C nginx --no-header|wc -l)
#判断Nginx是否存活
if [ $nginxpid -eq 0 ];then
systemctl stop keepalived
exit 1
fi
chmod -R 744 /usr/local/nginx/src/nginx_check.sh
配置说明:判断nginx是否存活,如果nginx挂了,就把keepalived停掉,备库的keepalived就会争抢主库的虚拟IP变成主库, 因为keepalived是服务器级别的监控,nginx挂掉并不会有什么反应,所以用脚本关闭keepalived让备库去抢占虚拟IP
5:启动keepalived
systemctl start keepalived
查看主服务器虚拟ip
查看主服务器广播
sudo tcpdump -i ens33 vrrp -n
通过keepalived的vrrp通信发布存活信息给备库
6:测试高可用
关闭主服务器nginx
systemctl stop nginx
监控日志tail -f /var/log/messages
发现异常:
1:Unsafe permissions found for script /usr/local/nginx/src/nginx_check.sh
解决:chmod -R 744 /usr/local/nginx/src/nginx_check.sh
2:default user 'keepalived_script' for script execution does not exist - please create.
解决:
global_def{
...
#添加配置
script_user root
enable_script_security
...
}
3: /usr/local/nginx/src/nginx_check.sh exited with status 1
解决:一般是由seliunx防火墙导致
setenforce 0 #临时关闭
vi /etc/sysconfig/selinux
SELINUX=disabled
4: /usr/local/nginx/src/nginx_check.sh exited due to signal 15
解决:vrrp_script{}中的interval时间需大于脚本中的sleep时间