Keepalived+Nginx四层代理 实现双负载均衡器高可用kube-apiserver
系统:Almalinux 9
架构:
192.168.100.10 masterA.k8s.local masterA
192.168.100.20 masterB.k8s.local masterB
192.168.100.30 masterC.k8s.local masterC
192.168.100.11 workA.k8s.local workA
192.168.100.22 workB.k8s.local workB
192.168.100.33 workC.k8s.local workC
192.168.100.100 LbA.k8s.local LbA
192.168.100.101 LbB.k8s.local LbB
现在你需要有一个kubernetes集群>>>《最新版Kubernetes部署教程v1.31.0》
基础配置(两台负载均衡器都需要配置)
# IP 地址参考hosts文件,或者文章顶部的架构
[root@LbA ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.10 masterA.k8s.local masterA
192.168.100.20 masterB.k8s.local masterB
192.168.100.30 masterC.k8s.local masterC
192.168.100.11 workA.k8s.local workA
192.168.100.22 workB.k8s.local workB
192.168.100.33 workC.k8s.local workC
192.168.100.100 LbA.k8s.local LbA # 负载均衡器A
192.168.100.101 LbB.k8s.local LbB # 负载均衡器B
# 关闭SElinux、防火墙
[root@LbA ~]# setenforce 0
[root@LbA ~]# vim /etc/selinux/config
...
SELINUX=disabled
...
[root@LbA ~]# systemctl disable --now firewalld
LbA安装并配置Keepalived
# 安装Keepalived
[root@LbA ~]# dnf install keepalived -y
# 配置Keepalived
[root@LbA ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LBA_MASTER # 主节点标识
}
vrrp_script chk_nginx {
script "/usr/local/bin/check_nginx.sh" # nginx状态检查脚本
interval 2 # 每隔2秒检查一次
weight -5 # 每次降低5权重
}
vrrp_instance VI_1 {
state MASTER # 主节点
interface ens160 # 绑定的物理网卡名
virtual_router_id 51 # 区分VRRP实例id,和备份节点一致
priority 101 # 权重
advert_int 1 # 设置广告包的发送间隔为 1 秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.100.99/24 # 虚拟IP
}
track_script {
chk_nginx
}
}
# nginx状态健康检查脚本
[root@LbA ~]# vim /usr/local/bin/check_nginx.sh
#!/bin/bash
NGINX_SERVICE="nginx" # nginx 服务名
KEEPALIVED_SERVICE="keepalived" # keepalived 服务名
RETRY_COUNT=2 # nginx尝试重启的次数
RETRY_DELAY=2 # nginx重启后等待的秒数
function check_nginx() { # 检查nginx服务状态函数
ps -C $NGINX_SERVICE --no-heading | wc -l
}
function log_message() { # 记录日志
echo "$(date +'%Y-%m-%d %H:%M:%S') - $1" >> /var/log/nginx_keepalived_check.log
}
counter=$(check_nginx) # 主逻辑
if [ "${counter}" -eq 0 ]; then
log_message "$NGINX_SERVICE is not running, attempting to start..."
systemctl start $NGINX_SERVICE
sleep $RETRY_DELAY
counter=$(check_nginx)
if [ "${counter}" -eq 0 ]; then
log_message "$NGINX_SERVICE failed to start, stopping $KEEPALIVED_SERVICE..."
systemctl stop $KEEPALIVED_SERVICE
else
log_message "$NGINX_SERVICE started successfully."
fi
else
log_message "$NGINX_SERVICE is running."
fi
# 防止日志过大启动日志轮转
[root@LbA ~]# vim /etc/logrotate.d/nginx_keepalived_check
/var/log/nginx_keepalived_check.log {
daily
rotate 1
missingok
notifempty
compress
delaycompress
postrotate
systemctl reload nginx > /dev/null 2>&1 || true
endscript
}
# 配置Keepalived自启动
[root@LbA ~]# systemctl enable --now keepalived
LbB安装并配置Keepalived
除了keepalived.conf不一样其他步骤都一样,LbB的keepalived.conf如下
[root@LbB ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LBB_BACKUP # 不同点
}
vrrp_script chk_nginx {
script "/usr/local/bin/check_nginx.sh"
interval 2
weight -5
}
vrrp_instance VI_1 {
state BACKUP # 不同点
interface ens160
virtual_router_id 51
priority 100 # 不同点
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.100.99/24
}
track_script {
chk_nginx
}
}
安装并配置Nginx(两节点配置一致)
# 添加最新版Nginx仓库源
[root@LbA ~]# vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
# 安装Nginx和四层代理模块
[root@LbA ~]# yum install nginx nginx-mod-stream -y
# 创建四层代理配置路径
[root@LbA ~]# mkdir -p /etc/nginx/stream.d
# 配置四层代理
[root@LbA ~]# vim /etc/nginx/nginx.conf
# 这是主配置文件不进行任何修改最后加上四层代理块,注意要和http块同级
...
http {
...
}
stream {
include /etc/nginx/stream.d/*.conf; # 四层代理配置文件位置
}
# 四层代理配置
[root@LbA ~]# vim /etc/nginx/stream.d/k8s-api.conf
# 三个控制平面
upstream k8s_api {
server masterA.k8s.local:6443;
server masterB.k8s.local:6443;
server masterC.k8s.local:6443;
}
server {
listen 6443; # 监听端口
proxy_pass k8s_api;
}
# 自启动Nginx
[root@LbA ~]# systemctl enable --now nginx
大家可以更改nginx配置文件把他改错停止服务并无法启动,查看keepalived服务是否配置正常,VIP是否正常漂移到LbB节点
现在把kubernete三个控制平面的hosts中的负载均衡器改成keepalived的VIP验证集群是否正常启动
[root@masterA ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.99 control-plane-endpoint.k8s.local control-plane-endpoint # 修改此IP
192.168.100.10 masterA.k8s.local masterA
192.168.100.20 masterB.k8s.local masterB
192.168.100.30 masterC.k8s.local masterC
192.168.100.11 workA.k8s.local workA
192.168.100.22 workB.k8s.local workB
192.168.100.33 workC.k8s.local workC
# 查看集群节点,可以看见可以正常获取到集群正常,状态为“NotReady”是因为我们网络组件还没有安装
[root@masterA ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
mastera.k8s.local NotReady control-plane 2d1h v1.31.0
masterb.k8s.local NotReady control-plane 2d1h v1.31.0
masterc.k8s.local NotReady control-plane 2d1h v1.31.0
worka.k8s.local NotReady <none> 2d v1.31.0
workb.k8s.local NotReady <none> 2d v1.31.0
workc.k8s.local NotReady <none> 2d v1.31.0