服务架构
两台Keepalived做主从热备,每个Keepalived节点都集成一台Nginx实现Nginx的HA,Nginx再反向代理Tomcat集群,同时实现Tomcat集群的负载均衡和动静分离。
服务分布
- 两个Keepalived主备节点
- 192.168.214.150
- 192.168.214.151
- 两个Nginx服务节点
- 192.168.214.150:80
- 192.168.214.151:80
- 三台Tomcat服务器
- 192.168.214.150:8080
- 192.168.214.151:8080
- 192.168.214.152:8080
概况
Keepalived: Load balancer and high availability service。这是官方对Keepalived的最直接的解释,Keepalived是一个负载均衡和高可用服务。我们在这里使用它的高可用特性,集成Nginx实现Nginx的高可用
前提
搭建Nginx+Tomcat集群架构:一台Nginx+三台Tomcat —— 大型网站分布式架构(八)—— Tomcat集群横向拓展 + Nginx负载均衡
###扩展Nginx服务
1.搭建第二台Nginx
之前我们已经在192.168.214.150节点上搭建了一台Nginx服务,现在需要在192.168.214.151节点上搭建另外一台,搭建方式相同:大型网站分布式架构(六)—— Nginx的安装
2.修改nginx.conf配置
两台Nginx的配置文件相同,都是设置了Tomcat集群的负载均衡,动静分离。所以直接分发192.168.214.150节点的nginx.conf给192.168.214.151节点即可
[root@centos6-1 ~]# cd /usr/local/nginx/conf/
[root@centos6-1 conf]# scp -r nginx.conf root@centos6-2:$PWD
nginx.conf 100% 913 0.9KB/s 00:00
[root@centos6-1 conf]#
搭建Keepalived主备
1.Keepalived安装
博主采用yum形式安装,省事简便,如果寻求版本的更新,可以使用源码编译安装。同时在两个节点上安装:
192.168.214.150 —— centos6-1
192.168.214.151 —— centos6-2
[root@centos6-1 ~]# yum install keepalived
[root@centos6-2 ~]# yum install keepalived
配置文件修改
配置文件所在位置:keepalived.conf
[root@centos6-1 ~]# cd /etc/keepalived/
[root@centos6-1 keepalived]# ll
total 4
-rw-r--r--. 1 root root 507 Aug 27 21:17 keepalived.conf
[root@centos6-1 keepalived]#
修改后192.168.214.150节点的keepalived.conf配置文件:
! Configuration File for keepalived
global_defs {
#keepalived所在主机名
router_id centos6-1
}
#检查nginx状态的执行脚本
vrrp_script kpld_ngx {
#脚本所在路径
script "/export/shell/kpld_ngx.sh"
interval 2
weight -15
}
vrrp_instance VI_1 {
#设置成备份节点(有MASTER/BACKUP两个值)
state BACKUP
#主机网卡名
interface eth0
#虚拟路由id
virtual_router_id 100
#所在主机ip
mcast_src_ip 192.168.214.150
#优先级
priority 100
advert_int 1
#认证
authentication {
auth_type PASS
auth_pass zaomianbao
}
#触发脚本
track_script {
kpld_ngx
}
#虚拟ip
virtual_ipaddress {
192.168.214.200
}
}
修改后192.168.214.151节点的keepalived.conf配置文件:
! Configuration File for keepalived
global_defs {
router_id centos6-2
}
vrrp_script kpld_ngx {
script "/export/shell/kpld_ngx.sh"
interval 2
weight -15
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 100
#所有配置就这里跟192.168.214.150节点不同,设置一个与节点一不同的值即可
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass zaomianbao
}
track_script {
kpld_ngx
}
virtual_ipaddress {
192.168.214.200
}
}
配置详解
配置项 | 解释 |
virtual_ipaddress | 虚拟ip,后面直接访问的ip |
router_id | keepalived所在主机名 |
interval | 执行间隔,单位秒 |
weight | 每次故障启动后减去此权重:priority-weight |
state | 都设置成BACKUP即可,也可以指定一台设置成MASTER |
interface | 网卡名,这个需要通过ifconfig或ip addr命令获取 |
virtual_router_id | 随意,只要保证两节点keepalived配置相同即可 |
priority | 优先级,优先级决定了主备身份,当都设置成BACKUP时,启动时谁优先级高谁是主节点 |
创建kpld_ngx.sh脚本
[root@centos6-1 ~]# cd /export/shell/
[root@centos6-1 shell]# touch kpld_ngx.sh
[root@centos6-1 shell]# chmod 777 kpld_ngx.sh
编写脚本内容
#!/bin/bash
COUNT=$(ps -C nginx --no-header |wc -l)
echo $COUNT
#判断Nginx 是否都挂掉了
if [ $COUNT -eq 0 ]
then
#如果挂掉了,就启动nginx
/usr/local/nginx/sbin/nginx
echo "重启nginx"
#等5秒钟后,再次查看是否 启动成功
sleep 5
#如果nginx没有启动起来,就直接干掉keepalived
COUNT=$(ps -C nginx --no-header |wc -l)
if [ $COUNT -eq 0 ]
then
echo "干掉keepalived"
killall keepalived
fi
fi
分发脚本给节点192.168.214.150
[root@centos6-1 shell]# scp kpld_ngx.sh root@centos6-2$PWD
启动各节点Tomcat
前提将项目appdemo部署到Tomcat上
[root@centos6-1 ~]# cd /export/servers/apache-tomcat-8.0.53/bin/
[root@centos6-1 bin]# ./startup.sh
[root@centos6-2 ~]# cd /export/servers/apache-tomcat-8.0.53/bin/
[root@centos6-2 bin]# ./startup.sh
[root@centos6-3 ~]# cd /export/servers/apache-tomcat-8.0.53/bin/
[root@centos6-3 bin]# ./startup.sh
访问Tomcat
192.168.214.150节点Tomcat
192.168.214.151节点Tomcat
192.168.214.152节点Tomcat
启动Nginx
两个节点的Nginx都启动
[root@centos6-1 ~]# cd /usr/local/nginx/sbin/
[root@centos6-1 sbin]# ./nginx
[root@centos6-2 ~]# cd /usr/local/nginx/sbin/
[root@centos6-2 sbin]# ./nginx
访问Nginx
访问192.168.214.150节点Nginx
访问192.168.214.151节点Nginx
节点二获取不到静态资源是因为,在节点二上没有部署静态资源,而节点一我们之前部署过
启动Keepalived
两个节点都启动
[root@centos6-1 ~]# service keepalived start
[root@centos6-2 ~]# service keepalived start
访问Keepalived虚拟ip
虚拟ip我们配置为192.168.214.200
发现访问成功
查看Keepalived的状态
查看两个节点Keepalived的日志
192.168.214.150节点Keepalived日志
Aug 29 16:34:32 centos6-1 Keepalived[2028]: Starting Keepalived v1.2.13 (03/19,2015)
Aug 29 16:34:32 centos6-1 Keepalived[2029]: Starting Healthcheck child process, pid=2030
Aug 29 16:34:32 centos6-1 Keepalived[2029]: Starting VRRP child process, pid=2031
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Netlink reflector reports IP 192.168.214.150 added
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Netlink reflector reports IP fe80::20c:29ff:fedf:1580 added
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Registering Kernel netlink reflector
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Registering Kernel netlink command channel
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Registering gratuitous ARP shared channel
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Truncating auth_pass to 8 characters
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Configuration is using : 63985 Bytes
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Using LinkWatch kernel netlink reflector...
#可以看到节点1为备份节点
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: VRRP_Instance(VI_1) Entering BACKUP STATE
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Netlink reflector reports IP 192.168.214.150 added
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Netlink reflector reports IP fe80::20c:29ff:fedf:1580 added
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Registering Kernel netlink reflector
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Registering Kernel netlink command channel
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Configuration is using : 6289 Bytes
Aug 29 16:34:32 centos6-1 kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Aug 29 16:34:32 centos6-1 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Aug 29 16:34:32 centos6-1 kernel: IPVS: ipvs loaded.
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: VRRP_Script(check_nginx) succeeded
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Using LinkWatch kernel netlink reflector...
Aug 29 16:34:36 centos6-1 Keepalived_vrrp[2031]: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 29 16:34:36 centos6-1 Keepalived_vrrp[2031]: VRRP_Instance(VI_1) Received higher prio advert
Aug 29 16:34:36 centos6-1 Keepalived_vrrp[2031]: VRRP_Instance(VI_1) Entering BACKUP STATE
192.168.214.151节点Keepalived日志
Aug 29 16:34:34 centos6-2 Keepalived[2381]: Starting Keepalived v1.2.13 (03/19,2015)
Aug 29 16:34:34 centos6-2 Keepalived[2382]: Starting Healthcheck child process, pid=2383
Aug 29 16:34:34 centos6-2 Keepalived[2382]: Starting VRRP child process, pid=2384
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Netlink reflector reports IP 192.168.214.151 added
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Netlink reflector reports IP fe80::250:56ff:fe31:7ab0 added
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Registering Kernel netlink reflector
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Registering Kernel netlink command channel
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Registering gratuitous ARP shared channel
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Truncating auth_pass to 8 characters
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Configuration is using : 63945 Bytes
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Using LinkWatch kernel netlink reflector...
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Entering BACKUP STATE
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Aug 29 16:34:34 centos6-2 kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Aug 29 16:34:34 centos6-2 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Aug 29 16:34:34 centos6-2 kernel: IPVS: ipvs loaded.
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: VRRP_Script(check_nginx) succeeded
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Netlink reflector reports IP 192.168.214.151 added
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Netlink reflector reports IP fe80::250:56ff:fe31:7ab0 added
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Registering Kernel netlink reflector
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Registering Kernel netlink command channel
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Configuration is using : 6249 Bytes
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Using LinkWatch kernel netlink reflector...
#可以看到节点2为主节点
Aug 29 16:34:36 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) forcing a new MASTER election
Aug 29 16:34:36 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) forcing a new MASTER election
Aug 29 16:34:37 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 29 16:34:38 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 29 16:34:38 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 29 16:34:38 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.200
Aug 29 16:34:38 centos6-2 Keepalived_healthcheckers[2383]: Netlink reflector reports IP 192.168.214.200 added
Aug 29 16:34:43 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.200
keepalived内部运行原理
- 我们可以随意停止任意节点的Keepalived服务,发现服务仍然可以使用,此时如果停止的节点是主节点,那么存活的备份节点会切换成为主节点提供服务,如果再启动停掉的节点,那么该节点会自动成为备份节点。
- 我们也可以停掉任意节点的Nginx服务,当Nginx服务停掉之后,我们的检测Nginx状态的脚本会主动启动停掉的Nginx。
[root@centos6-2 ~]# ps -ef|grep nginx
root 6790 1 0 17:00 ? 00:00:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody 6792 6790 0 17:00 ? 00:00:00 nginx: worker process
root 6899 2262 0 17:01 pts/0 00:00:00 grep nginx
[root@centos6-2 ~]# /usr/local/nginx/sbin/nginx -s stop
[root@centos6-2 ~]# ps -ef|grep nginx
root 6922 6921 0 17:01 ? 00:00:00 /bin/bash /export/shell/kpld_ngx.sh
root 6927 1 0 17:01 ? 00:00:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody 6929 6927 0 17:01 ? 00:00:00 nginx: worker process
root 6931 2262 0 17:01 pts/0 00:00:00 grep nginx
[root@centos6-2 ~]#
可以看到kpld_ngx.sh脚本在Nginx停掉后主动运行,又重新启动了,因为我们发现Nginx进程号发生了变化。
总结
到此我们就真正实现了网站服务的高可用。当然,我们还可以考虑一下优化的问题,比如访问到了节点2的Nginx服务时由于没有本地静态资源导致了无法获得对应的静态的资源,那么我们就可以把静态摘取出来搭建一个独立的静态资源系统。我们的项目并没有接入数据库,业务中也没有数据逻辑的处理,当计算达到一个顶峰时,当与数据库的连接达到一个极限时,很多很多的优化存在在项目开发的边边角角,而社区中都会有很多成熟的解决方案。博主将该专栏就先更到这个阶段,后期如果有机缘的话,博主会深入到微服务再到数据库等等方面。祝大家都能成为优秀的程序员。
参考:
http://www.linuxde.net/2013/04/13381.html
http://www.keepalived.org/manpage.html
https://blog.csdn.net/yelllowcong/article/details/78764780
https://www.cnblogs.com/youzhibing/p/7327342.html
https://www.jianshu.com/p/b050d8861fc1
https://baike.baidu.com/item/Keepalived/10346758?fr=aladdin
---------------------
作者:枣面包
来源:CSDN
原文:https://blog.csdn.net/weixin_37490221/article/details/82178762
版权声明:本文为博主原创文章,转载请附上博文链接!