Linux集群概述
- 根据功能划分为两大类:高可用和负载均衡
- 高可用集群通常为两台服务器,一台工作,另外一台作为冗余,当提供服务的机器宕机,冗余将接替继续提供服务
- 实现高可用的开源软件有:heartbeat、keepalived
- 负载均衡集群,需要有一台服务器作为分发器,它负责把用户的请求分发给后端的服务器处理,在这个集群里,除了分发器外,就是给用户提供服务的服务器了,这些服务器数量至少为2
- 实现负载均衡的开源软件有LVS、keepalived、haproxy、nginx,商业的有F5、Netscaler
keepalived介绍
- 在这里我们使用keepalived来实现高可用集群,因为heartbeat在centos6上有一些问题,影响实验效果
- keepalived通过VRRP(Virtual Router Redundancy Protocl)来实现高可用。
- 在这个协议里会将多台功能相同的路由器组成一个小组,这个小组里会有1个master角色和N(N>=1)个backup角色。
- master会通过组播的形式向各个backup发送VRRP协议的数据包,当backup收不到master发来的VRRP数据包时,就会认为master宕机了。此时就需要根据各个backup的优先级来决定谁成为新的mater。
- Keepalived要有三个模块,分别是core、check和vrrp。其中core模块为keepalived的核心,负责主进程的启动、维护以及全局配置文件的加载和解析,check模块负责健康检查,vrrp模块是来实现VRRP协议的。
用keepalived配置高可用
- 配置是针对nginx服务,两台服务器分别安装nginx和keepalived,实现master机上的nginx服务不正常,启动backup机上的nginx;
- 对外使用VIP实现服务,keepalived根据机器情况(脚本配置)把VIP配置给其中一个服务器;
- 测试:两个服务器的真实IP显示不同内容;连接VIP显示的内容;master出现问题时,连接VIP显示的内容;master恢复,连接VIP显示的内容;相关的日志信息;
- 主机MASTER配置
- 安装nginx,可以yum安装,也可以编译安装,安装后实现以下状态
[root@draft ~]# curl -x127.0.0.1:8080 localhost #编缉虚拟主机文件,显示如下信息;
master server in use.
- 安装和配置keepalived
yum install -y keepalived
vi /etc/keepalived/keepalived.conf #写入以下内容;
global_defs {
notification_email { #这一部分用于发送通知邮件,可忽略;
aming@aminglinux.com
}
notification_email_from root@aminglinux.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_nginx {
script "/usr/local/sbin/check_ng.sh" #检查服务是否启动的脚本;
interval 3 #检测间隔;
}
vrrp_instance VI_1 { #定义一个虚拟路由;
state MASTER
#当前节点在此此虚拟路由器上的初始状态,只能有一个是MASTER,余下的都应该为BACKUP,启动后根据priority竞选;
interface ens33 #监听的实际网口;
virtual_router_id 51 #组播ID;
priority 100 #主机即本机的权重;
advert_int 1 #发送组播包的间隔时间,默认为1秒;
authentication { #认证配置;
auth_type PASS
auth_pass aminglinux>com
}
virtual_ipaddress { #指定漂移地址;
192.168.87.222
}
track_script {
chk_nginx
}
}
vi /usr/local/sbin/check_ng.sh #写入以下内容;
#!/bin/bash
#时间变量,用于记录日志
d=`date --date today +%Y%m%d_%H:%M:%S`
#计算nginx进程数量
n=`ps -C nginx --no-heading|wc -l`
#如果进程为0,则启动nginx,并且再次检测nginx进程数量,
#如果还为0,说明nginx无法启动,此时需要关闭keepalived,释放VIP,同时所有BACKUP就收不到MASTER的信息,BACKUP根据优先级获取VIP,提供服务;
if [ $n -eq "0" ]; then
systemctl start nginx #安装方式不一样,启动方式不一样;
n2=`ps -C nginx --no-heading|wc -l`
if [ $n2 -eq "0" ]; then
echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
systemctl stop keepalived
fi
fi
chmod 755 /usr/local/sbin/check_ng.sh #要修改脚本权限,不然脚本不能使用;
systemctl start keepalived #启动服务;
- 单机检查:
检查服务进程:
[root@draft ~]# ps aux |grep keepalived
root 22501 0.0 0.1 122888 1412 ? Ss 16:04 0:00 /usr/sbin/keepalived -D
root 22502 0.0 0.3 133848 3340 ? S 16:04 0:00 /usr/sbin/keepalived -D
root 22503 0.0 0.2 133788 2900 ? S 16:04 0:00 /usr/sbin/keepalived -D
root 25218 0.0 0.0 112728 980 pts/0 S+ 16:27 0:00 grep --color=auto keepalived
[root@draft ~]# ps aux |grep nginx #关闭也会由keepalived启动;
root 22417 0.0 0.0 46344 972 ? Ss 16:04 0:00 nginx: master process nginx
nginx 22421 0.0 0.2 46752 2168 ? S 16:04 0:00 nginx: worker process
root 25236 0.0 0.0 112728 968 pts/0 R+ 16:27 0:00 grep --color=auto nginx
[root@draft ~]# curl -x192.168.87.222:8080 192.168.87.222
master server in use.
- 从机BACKUP配置
- 安装nginx,可以yum安装,也可以编译安装,安装后实现以下状态
[root@second ~]# curl -x127.0.0.1:8080 localhost #编缉虚拟主机文件,显示如下信息;
back up server in use.
- 安装和配置keepalived
yum install -y keepalived
vi /etc/keepalived/keepalived.conf #写入以下内容;
global_defs {
notification_email { #这一部分用于发送通知邮件,可忽略;
aming@aminglinux.com
}
notification_email_from root@aminglinux.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_nginx {
script "/usr/local/sbin/check_ng.sh" #检查服务是否启动的脚本;
interval 3 #检测间隔;
}
vrrp_instance VI_1 { #定义一个虚拟路由;
state BACKUP
#当前节点在此此虚拟路由器上的初始状态,只能有一个是MASTER,余下的都应该为BACKUP,启动后根据priority竞选;
interface ens33 #监听的实际网口;
virtual_router_id 51 #组播ID;
priority 90 #从机即本机的权重;
advert_int 1 #发送组播包的间隔时间,默认为1秒;
authentication { #认证配置;
auth_type PASS
auth_pass aminglinux>com
}
virtual_ipaddress { #指定漂移地址;
192.168.87.222
}
track_script {
chk_nginx
}
}
vi /usr/local/sbin/check_ng.sh #跟主机的检测脚本一样;
chmod 755 /usr/local/sbin/check_ng.sh #要修改脚本权限,不然脚本不能使用;
systemctl start keepalived #启动服务;
- 单机检查:
检查服务进程:
[root@second ~]# ps aux | grep keepalived
root 32973 0.0 0.1 122888 1408 ? Ss 16:40 0:00 /usr/sbin/keepalived -D
root 32974 0.0 0.2 127088 2420 ? S 16:40 0:00 /usr/sbin/keepalived -D
root 32975 0.0 0.2 127088 2452 ? S 16:40 0:00 /usr/sbin/keepalived -D
root 32991 0.0 0.0 112728 980 pts/1 R+ 16:40 0:00 grep --color=auto keepalived
[root@second ~]# ps aux | grep nginx
root 32928 0.0 0.0 46344 976 ? Ss 16:21 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx 32929 0.0 0.2 46752 2168 ? S 16:21 0:00 nginx: worker process
root 32993 0.0 0.0 112728 968 pts/1 R+ 16:40 0:00 grep --color=auto nginx
[root@second ~]# pkill nginx
[root@second ~]# ps aux | grep nginx #杀死后,会再次启动nginx;
root 33010 0.0 0.0 46344 968 ? Ss 16:40 0:00 nginx: master process nginx
nginx 33014 0.0 0.1 46752 1924 ? S 16:40 0:00 nginx: worker process
root 33016 0.0 0.0 112728 972 pts/1 R+ 16:40 0:00 grep --color=auto nginx
[root@second ~]# curl -x192.168.87.222:8080 192.168.87.222 #从机上访问VIP;
master server in use.
- 将nginx程序转移位置后,检测脚本的日志情况
[root@second ~]# mv /usr/sbin/nginx /usr/sbin/nginx.bak
[root@second ~]# pkill nginx #杀死后,keepalived不能再将nginx启动,脚本停止keepalived;
[root@second ~]# tail /var/log/check_ng.log
20191107_16:49:06 nginx down,keepalived will stop
综合测试
[root@draft ~]# systemctl stop keepalived #MASTER停止keepalived,BACKUP将会提供服务;
[root@draft ~]# !curl
curl -x192.168.87.222:8080 192.168.87.222
back up server in use.
[root@second ~]# ip addr
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:97:47:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.87.150/24 brd 192.168.87.255 scope global dynamic ens33
valid_lft 1519sec preferred_lft 1519sec
inet 192.168.87.222/32 scope global ens33 #VIP在BACKUP中使用;
valid_lft forever preferred_lft forever
inet6 fe80::50a5:f454:5189:9942/64 scope link noprefixroute
valid_lft forever preferred_lft forever
MASTER日志:
Nov 7 16:52:42 draft Keepalived[22501]: Stopping #停止服务的细节;
Nov 7 16:52:42 draft systemd: Stopping LVS and VRRP High Availability Monitor...
Nov 7 16:52:42 draft Keepalived_vrrp[22503]: VRRP_Instance(VI_1) sent 0 priority
Nov 7 16:52:42 draft Keepalived_vrrp[22503]: VRRP_Instance(VI_1) removing protocol VIPs. #释放VIP;
Nov 7 16:52:42 draft Keepalived_healthcheckers[22502]: Stopped
Nov 7 16:52:43 draft Keepalived_vrrp[22503]: Stopped
Nov 7 16:52:43 draft Keepalived[22501]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Nov 7 16:52:43 draft systemd: Stopped LVS and VRRP High Availability Monitor.
BACKUP日志:
Nov 7 16:52:42 draft Keepalived_vrrp[34156]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 7 16:52:43 draft Keepalived_vrrp[34156]: VRRP_Instance(VI_1) Entering MASTER STATE #BACKUP进入服务状态;
Nov 7 16:52:43 draft Keepalived_vrrp[34156]: VRRP_Instance(VI_1) setting protocol VIPs. #设置VIPs;
Nov 7 16:52:43 draft Keepalived_vrrp[34156]: Sending gratuitous ARP on ens33 for 192.168.87.222
Nov 7 16:52:43 draft Keepalived_vrrp[34156]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.87.222
Nov 7 16:52:43 draft Keepalived_vrrp[34156]: Sending gratuitous ARP on ens33 for 192.168.87.222
[root@draft ~]# systemctl start keepalived #MASTER恢复服务;
[root@draft ~]# curl -x192.168.87.222:8080 192.168.87.222
master server in use.
[root@draft ~]# ip addr
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:bd:e9:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.87.149/24 brd 192.168.87.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 192.168.87.222/32 scope global ens33 #VIP在MASTER中使用;
valid_lft forever preferred_lft forever
inet6 fe80::d8da:c84:7947:a438/64 scope link noprefixroute
valid_lft forever preferred_lft forever
MASTER日志:
Nov 7 17:03:50 draft systemd: Starting LVS and VRRP High Availability Monitor... #启动过程...
Nov 7 17:03:50 draft Keepalived[28407]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Nov 7 17:03:50 draft Keepalived[28407]: Opening file '/etc/keepalived/keepalived.conf'.
Nov 7 17:03:50 draft systemd: PID file /var/run/keepalived.pid not readable (yet?) after start.
Nov 7 17:03:50 draft Keepalived[28408]: Starting Healthcheck child process, pid=28409
Nov 7 17:03:50 draft Keepalived[28408]: Starting VRRP child process, pid=28410
Nov 7 17:03:50 draft systemd: Started LVS and VRRP High Availability Monitor.
Nov 7 17:03:50 draft Keepalived_healthcheckers[28409]: Opening file '/etc/keepalived/keepalived.conf'.
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: Registering Kernel netlink reflector
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: Registering Kernel netlink command channel
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: Registering gratuitous ARP shared channel
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: Opening file '/etc/keepalived/keepalived.conf'.
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: Truncating auth_pass to 8 characters
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: Using LinkWatch kernel netlink reflector...
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
接上
Nov 7 17:03:50 draft Keepalived_vrrp[28410]: VRRP_Script(chk_nginx) succeeded
Nov 7 17:03:51 draft Keepalived_vrrp[28410]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 7 17:03:52 draft Keepalived_vrrp[28410]: VRRP_Instance(VI_1) Entering MASTER STATE #进入MASTER状态;
Nov 7 17:03:52 draft Keepalived_vrrp[28410]: VRRP_Instance(VI_1) setting protocol VIPs. #设置VIPs;
Nov 7 17:03:52 draft Keepalived_vrrp[28410]: Sending gratuitous ARP on ens33 for 192.168.87.222
Nov 7 17:03:52 draft Keepalived_vrrp[28410]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.87.222
Nov 7 17:03:52 draft Keepalived_vrrp[28410]: Sending gratuitous ARP on ens33 for 192.168.87.222
BACKUP日志:
Nov 7 17:03:51 draft Keepalived_vrrp[34156]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 90 #收到更高优先级的信息;
Nov 7 17:03:51 draft Keepalived_vrrp[34156]: VRRP_Instance(VI_1) Entering BACKUP STATE #进入BACKUP状态;
Nov 7 17:03:51 draft Keepalived_vrrp[34156]: VRRP_Instance(VI_1) removing protocol VIPs.
- 建议重装服务后,本例是nginx,要重装keepalived,可能会检查在用的服务;重装nginx后,不重装keepalived使用不正常;
- 脑裂
脑裂是因为cluster分裂导致的,cluster集群中节点因为处理器忙或者其他原因暂时停止响应时,其他节点可能误认为该节点“已死”,从而夺取共享磁盘(即资源)的访问权,此时极有可能假死节点重新对共享文件系统产生读写操作,从而导致共享磁盘文件系统损坏。