一、前言
本文主要在两台服务器上搭建Nginx与keepalived,实现一个高可用集群的简易版本。
节点信息如下:
node1 192.168.100.133 Nginx、Keepalived(主)
node3 192.168.100.131 Nginx、Keepalived(备)
需要在两台服务器上安装Nginx,Keepalived。
安装配置Nginx比较简单,参考:centos7安装配置Nginx
切记,最好不要使用指令yum直接安装!!!网传这是Keepalived自身的问题,采用指令直接安装,配置 vrrp_script脚本时会报错类似如下:
Failed to dynamic link an ipset function - /lib64/libipset.so: undefined symbol: ipset_session_error。
本人校验后亦出现此类问题,故放弃yum命令安装,采用从官网下载包解压编译安装。对于已经yum指令安装的,可以先使用yum remove keepalived指令卸载。
二、安装配置
yum install ipvsadm popt-devel openssl-devel libnfnetlink-devel
1.下载
cd /usr/local/src
wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz
2.解压
tar -zxvf keepalived-1.3.5
3.安装
cd keepalived-1.3.5/ && ./configure --prefix=/usr/local/keepalived
4.编译
make && make install
5.为了方便统一管理,将解压编译好的keepalived 配置文件进行地方修改。
mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/init.d/
cp /usr/local/src/keepalived-1.3.5/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
ln -s /usr/local/keepalived/sbin/keepalived /sbin/
6.设置开机启动:
chkconfig keepalived on
三、Nginx监控脚本与keepalived配置文件
vi /usr/local/common/nginx-keepalived-related/check_nginx.log
1.新建nginx 检测重启脚本(主备节点上均设置):
vi /usr/local/common/nginx-keepalived-related/check_nginx.sh
为监控脚本没有执行权限:
chmod +x /usr/local/common/nginx-keepalived-related/check_nginx.sh
脚本文件内容如下。
#!/bin/bash
#定义时间变量,用于纪录日志
d=`date --date today +%Y%m%d_%H:%M:%S`
#计算nginx的进程数量
n=`ps -C nginx --no-heading|wc -l`
#如果n为0,则启动nginx,并在次检测nginx,还是为0说明nginx无法启动,则关闭Keepalived
if [ $n -eq "0" ]; then
# 根据Nginx启动方式配置。
# systemctl start nginx
/usr/local/nginx/sbin/nginx
echo "$d nginx 启动..." >> /usr/local/common/nginx-keepalived-related/check_nginx.log
n2=`ps -C nginx --no-heading|wc -l`
if [ $n2 -eq "0" ]; then
echo "$d nginx,keepalived will stop" >> /usr/local/common/nginx-keepalived-related/check_nginx.log
systemctl stop keepalived
fi
fi
2. keepalived最主要的是掌握它的配置文件,主配置文件内容如下(主节点上设置):
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
vrrp_script chk_nginx {
script "/usr/local/common/nginx-keepalived-related/check_nginx.sh" # 运行nginx 检测重启脚本
interval 10 # 检测脚本执行的间隔,可自定义,此处为了测试设置较大值。
}
# 定义虚拟路由,VI_1 为虚拟路由的标示符,自定义
vrrp_instance VI_1 {
state MASTER # 状态只有MASTER和BACKUP两种,并且要大写,MASTER工作态 BACKUP备用态
interface ens33 # 网卡,通过ens33广播,根据自己的机器填写
virtual_router_id 51 # 路由标识。主备机必须相同
mcast_src_ip 192.168.100.133 # 填写本机ip
priority 100 # 优先级。 值范围 0-254,master优先级必须必backup高
nopreempt # 优先级高的设置 nopreempt 解决异常恢复后再次抢占的问题
advert_int 1 # 组播信息发送间隔,两个节点设置必须一样,默认 1s
authentication {
auth_type PASS
auth_pass 1111
}
# 将 track_script 块加入 VI_1 配置块
track_script {
chk_nginx # 执行 Nginx 监控的服务
}
# 虚拟 IP 池, 两个节点设置必须一样
virtual_ipaddress {
192.168.100.120 # 虚拟ip,通过此虚拟IP访问主备机上的服务,也可扩展,可配置多个。
}
}
3. 备节点上的配置文件如下:(备节点上设置)
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
vrrp_script chk_nginx {
script "/usr/local/common/nginx-keepalived-related/check_nginx.sh" # 运行nginx 检测重启脚本
interval 10 # 检测脚本执行的间隔,可自定义,此处为了测试设置较大值。
}
# 定义虚拟路由,VI_1 为虚拟路由的标示符,自定义
vrrp_instance VI_1 {
state BACKUP # 状态只有MASTER和BACKUP两种,并且要大写,MASTER工作态 BACKUP备用态
interface ens33 # 网卡,通过ens33广播,根据自己的机器填写
virtual_router_id 51 # 路由标识。主备机必须相同
mcast_src_ip 192.168.100.131 # 填写本机ip
priority 90 # 优先级。值范围 0-254,master优先级必须必backup高
advert_int 1 # 组播信息发送间隔,两个节点设置必须一样,默认 1s
authentication {
auth_type PASS
auth_pass 1111
}
# 将 track_script 块加入 instance 配置块
track_script {
chk_nginx # 执行 Nginx 监控的服务
}
# 虚拟 IP 池, 两个节点设置必须一样
virtual_ipaddress {
192.168.100.120 # 虚拟ip,通过此虚拟IP访问主备机上的服务,也可扩展,可配置多个。
}
}
4. 修改 Nginx 欢迎首页内容(用于后面测试, 用于区分两个节点的 Nginx):
# vi /usr/local/nginx/html/index.html
192.168.100.133 中的标题加 node1...
<h1>Welcome to nginx! node1...</h1>
192.168.100.131 中的标题加 node3...
<h1>Welcome to nginx! node3...</h1>
四、测试
0.打开日志监控keepalived运行情况:
# 查看Keepalived 启动运行日志
tail -f -n100 /var/log/messages
注:messages中关于keepalived的启动失败或异常有多中情况,包括wearing警告,error错误等,需要仔细观察日志输出内容,具体问题参考最后的处理方法。
1.keepalived常用指令
# 启动
system start keepalived
# 停止
system stop keepalived
# 查看当前状态
systemctl status keepalived
2.查看keepalived进程
两台服务器启动好以后,可以通过ip a 指令查看到当前主机192.168.100.133的ip地址中包含的虚拟ip信息,如下:
此时的备机192.168.100.131的IP地址中没有虚拟IP信息,如下:
此时两个虚拟机中的Nginx都已经启动,如下:
此时,模拟node1节点宕机,挂起此虚拟机,再次查看另一个节点的ip信息,发现虚拟ip已经“漂移”到了当前节点:
再次刷新访问虚拟ip的页面,发现此时Nginx已经切换到了另一个节点上:
至此,完成keepalived + Nginx 的高可用集群搭建。
五、总结
1.对于整个搭建过程而言,确实存在一些坑的地方,比如yum指令直接安装后运行异常;配置文件或者进程文件修改后,就算使用重载命令,也要先关闭所有keepalived相关的进程等等。
2.keepalived运行的日志对于新手来说可能不算友好,提示的警告或提示等都有可能导致keepalived运行失败,但是此时查看进程确是正常的三个进程,看不出什么问题,需要执行systemctl status keepalived查看运行状态。
3.动手实践+看日志定位问题,是解决自身问题的最好办法。百度到的都是作为参考。
六、问题集锦
问题一、Keepalived 启动报错 PID file /var/run/keepalived.pid not readable (yet?) after start.
解决:
1.先杀死所有与keepalived相关的进程
ps ax|grep keepalived
kill -9 进程号
2.修改pid进程文件
vi /lib/systemd/system/keepalived.service
PIDFile=/var/run/keepalived.pid
3.重新载入配置
systemctl daemon-reload
4.重启keepalived
systemctl start keepalived
问题二、Can‘t open PID file /var/run/keepalived.pid (yet?) after start: No such file or directory
使用pkill keepalived杀死所有与keepalived相关的进程,再重启keepalived
参考网址:https://blog.csdn.net/linux_yyp/article/details/107181466
问题三、keepalived日志提示:WARNING - script '/usr/local/common/nginx-keepalived-related/check_nginx.sh' is not executable for uid:gid 0:0 - disabling.
原因:keepalived监控脚本没有执行权限
解决:chmod +x /usr/local/common/nginx-keepalived-related/check_nginx.sh
七、参考网址
https://blog.csdn.net/xiaozhegaa/article/details/106178304