HAProxy+Keepalived 负载均衡高可用配置
准备工作
五台虚拟机
主机 | IP | 功能 |
---|---|---|
主调度器 | 192.168.188.10 | 主要调度器,转发漂移地址 |
从调度器 | 192.168.188.20 | 备份调度器,起主调度器的备份作用 |
节点服务器1-3 | 192.168.188.100-102 | web服务器 |
安装haproxy和keepalived
编译安装HAProxy
详细安装请看HAProxy 搭建Web群集.
curl -R -O https://www.lua.org/ftp/lua-5.4.3.tar.gz
tar xf lua-5.4.3.tar.gz -C /usr/local
cd /usr/local/lua-5.4.3
yum install -y readline-devel
make linux
查看安装版本
./src/lua
lua-5.4.3 Copyright (C) 1994-2020 Lua.org, PUC-Rio
> print('hello world')
hello world
yum install -y gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel net-tools iotop bc zlib-devel ntpdate lsof tcpdump
进入haproxy源码包目录
make -j `lscpu |awk 'NR==4{print $2}'` ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE
_ZLIB=1 USE_SYSTEMD=1 USE_CPU_AFFINITY=1 USE_LUA=1 LUA_INC=/usr/local/lua-5.4.3/src/ LUA_LIB=/usr/local/lua-5.4.3/src/ PREFIX=/usr/local/haproxy
echo $? 检查是否成功编译
make install PREFIX=/usr/local/haproxy
cd /usr/local/haproxy/sbin
./haproxy -v
HA-Proxy version 2.3.5-5902ad9 2021/02/06 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2022.
Known bugs: http://www.haproxy.org/bugs/bugs-2.3.5.html
Running on: Linux 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64
启动haproxy需要的配置
HAProxy启动脚本
vim /lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target
[Service]
ExecStartPre=/usr/local/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/local/haproxy/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
[Install]
WantedBy=multi-user.target
创建配置文件目录
mkdir -p /etc/haproxy
将样本配置文件拷贝到/etc/haproxy里
cp examples/haproxy.cfg /etc/haproxy/haproxy.cfg
如果没有haproxy.cfg 可以自己写一个
vim /etc/haproxy/haproxy.cfg
global
user haproxy # 用户
group haproxy
daemon
nbproc 2
#cpu-map 1 0
#cpu-map 2 1
maxconn 100000
chroot /usr/local/haproxy # 锁定家目录
pidfile /var/lib/haproxy/haproxy.pid #pid文件位置
log 127.0.0.1 local0 info
defaults
log global
option httplog
option http-keep-alive
option redispatch
option forwardfor
maxconn 100000
mode http
retries 3
timeout check 5s
timeout connect 5s
timeout client 60s
timeout server 60s
timeout http-request 10s
timeout queue 1m
listen stats
bind 0.0.0.0:8888 # 端口8888
log global
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /stats # 查看状态网页后缀
stats refresh 5s
stats auth admin:123 # 授权访问 用户名:密码
创建haproxy用户和组
groupadd haproxy
useradd -M -s /sbin/nologin haproxy -g haproxy
给用户haproxy授权
mkdir -p /var/lib/haproxy
chown -R haproxy:haproxy /usr/local/haproxy/
chown -R haproxy:haproxy /var/lib/haproxy/
启动haproxy
systemctl start haproxy
systemctl status haproxy
浏览器验证 输入ip:端口号/stats
两台调度器的haproxy都安装成功
安装Keepalived
yum install -y keepalived
节点服务器安装nginx和写入网页
yum install -y nginx
cd /usr/share/nginx/html
echo 'this is web100' > ./index.html
systemctl start nginx
配置haproxy
主副调度器haproxy配置可以一样
vim /etc/haproxy/haproxy.cfg
global
user haproxy
group haproxy
daemon
nbproc 2
#cpu-map 1 0
#cpu-map 2 1
maxconn 100000
#chroot /usr/local/haproxy
#pidfile /var/lib/haproxy/haproxy.pid
log 127.0.0.1 local2 info
defaults
log global
option httplog
#option http-server-close
option dontlognull
maxconn 100000
mode http
retries 3
timeout connect 5000
timeout client 50000
timeout server 50000
listen stats
bind 0.0.0.0:8888
log global
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /stats
stats refresh 5s
stats auth admin:123
listen WEB_PORT_80
bind 192.168.188.188:80
mode http
option httpchk GET /index.html
balance roundrobin
server web1 192.168.188.150:80 check inter 2000 fall 3 rise 5
server web2 192.168.188.101:80 check inter 2000 fall 3 rise 5
server web3 192.168.188.102:80 check inter 2000 fall 3 rise 5
从服务器启动HAProxy的时候可能会启动不了
因为监听了漂移ip 但是备服务器上没漂移ip
解决方法:
vi /etc/sysctl.conf
添加
net.ipv4.ip_nonlocal_bind = 1 # 忽略监听ip的检查
sysctl -p
配置keepalived
主调度器配置keepalived
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id HAP1
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 66
priority 120
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.188.188
}
}
副调度器配置keepalived
! Configuration File for keepalived
global_defs {
router_id HAP2 # 组名字修改
}
vrrp_instance VI_1 {
state BACKUP # 改成备份
interface ens33
virtual_router_id 66
priority 100 # 优先级比主调度器低
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.188.188
}
}
ip addr
查看虚拟地址
主调度器获得虚拟地址
副调度器没有虚拟地址
配置keepalived自动切换脚本
编写主调度器的自动检测脚本
#!/bin/bash
a=`ps -C haproxy --no-header|wc -l`
if [ $a -eq 0 ];then
systemctl start haproxy
echo "haproxy start..."
sleep 3
if [ `ps -C haproxy --no-header|wc -l` -eq 0 ];then
systemctl stop keepalived
echo "haproxy is down"
sleep 3
fi
fi
在主调度器的keepalived.conf文件中添加
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id HAP1
}
vrrp_script check_haproxy { # 自动检测脚本的方案名称
script '/etc/keepalived/check_haproxy.sh' # 脚本的绝对路径
interval 2 # 自动运行的间隔
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 66
priority 120
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.188.188
}
track_script { # 追踪脚本
check_haproxy # 追踪的脚本方案名称
}
}
chmod +x check_haproxy.sh
systemctl enable keepalived --now
systemctl restart keepalived
编写副调度器的自动检测脚本
#!/bin/bash
a=`ip a | grep 192.168.188.188 | wc -l`
b=`ps -ef | grep haproxy | grep -v grep | awk '{print $2}'`
if [ $a -gt 0 ];then
systemctl start haproxy
else
kill -9 $b
sleep 3600
fi
! Configuration File for keepalived
global_defs {
router_id HAP2
}
vrrp_script check_haproxy {
script '/etc/keepalived/check_haproxy.sh'
interval 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 66
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.188.188
}
track_script {
check_haproxy
}
}
chmod +x check_haproxy.sh
systemctl enable keepalived --now
systemctl restart keepalived
设置HAProxy的日志
查看haproxy的主配置文件
vim /etc/haproxy/haproxy.cfg
在global下面添加有
log 127.0.0.1 local2 info # 这里的local2可以是{1..7之间}
然后下面defaults里面添加有
log global
vim /etc/rsyslog.conf
添加
local2.* /var/log/haproxy.log
将注释取消
$ModLoad imudp
$UDPServerRun 514
保存之后重启服务
systemctl restart haproxy
systemctl restart rsyslog
systemctl status rsyslog
然后就会出现haproxy的日志文件
ls /var/log/haproxy.log
测试
测试负载均衡
打开浏览器
这是主调度器的状态界面
这是副调度器的状态界面
连接数都为0 因为是副调度器
刷新浏览器
轮询成功!!!!!!!
测试高可用
我们将主调度器关闭
继续刷新网页
并且动态查看日志
tailf /var/log/messages
副调度器成为主调度器
刷新网页之后 网页也没受影响 副调度器的状态界面也产生了信息
最后我们又把主调度器开启
漂移地址回到主调度器