Keepalived+Nginx 实现双负载均衡器高可用kube-apiserver

Keepalived+Nginx四层代理 实现双负载均衡器高可用kube-apiserver

系统：Almalinux 9
架构：
192.168.100.10 masterA.k8s.local masterA
192.168.100.20 masterB.k8s.local masterB
192.168.100.30 masterC.k8s.local masterC
192.168.100.11 workA.k8s.local workA
192.168.100.22 workB.k8s.local workB
192.168.100.33 workC.k8s.local workC
192.168.100.100 LbA.k8s.local LbA
192.168.100.101 LbB.k8s.local LbB

现在你需要有一个kubernetes集群>>>《最新版Kubernetes部署教程v1.31.0》

基础配置（两台负载均衡器都需要配置）

# IP 地址参考hosts文件，或者文章顶部的架构
[root@LbA ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.10 masterA.k8s.local masterA
192.168.100.20 masterB.k8s.local masterB
192.168.100.30 masterC.k8s.local masterC
192.168.100.11 workA.k8s.local workA
192.168.100.22 workB.k8s.local workB
192.168.100.33 workC.k8s.local workC
192.168.100.100 LbA.k8s.local LbA	# 负载均衡器A
192.168.100.101 LbB.k8s.local LbB	# 负载均衡器B

# 关闭SElinux、防火墙
[root@LbA ~]# setenforce 0
[root@LbA ~]# vim /etc/selinux/config
...
SELINUX=disabled
...
[root@LbA ~]# systemctl disable --now firewalld

LbA安装并配置Keepalived

# 安装Keepalived
[root@LbA ~]# dnf install keepalived -y
# 配置Keepalived
[root@LbA ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LBA_MASTER	# 主节点标识
}

vrrp_script chk_nginx {
    script "/usr/local/bin/check_nginx.sh"	# nginx状态检查脚本
    interval 2	# 每隔2秒检查一次
    weight -5	# 每次降低5权重
}

vrrp_instance VI_1 {
    state MASTER	# 主节点
    interface ens160	# 绑定的物理网卡名
    virtual_router_id 51	# 区分VRRP实例id，和备份节点一致
    priority 101	# 权重
    advert_int 1	# 设置广告包的发送间隔为 1 秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.100.99/24	# 虚拟IP
    }
    track_script {
        chk_nginx
    }
}

# nginx状态健康检查脚本
[root@LbA ~]# vim /usr/local/bin/check_nginx.sh
#!/bin/bash

NGINX_SERVICE="nginx"   # nginx 服务名
KEEPALIVED_SERVICE="keepalived" # keepalived 服务名
RETRY_COUNT=2   # nginx尝试重启的次数
RETRY_DELAY=2   # nginx重启后等待的秒数

function check_nginx() {        # 检查nginx服务状态函数
    ps -C $NGINX_SERVICE --no-heading | wc -l
}

function log_message() {        # 记录日志
    echo "$(date +'%Y-%m-%d %H:%M:%S') - $1" >> /var/log/nginx_keepalived_check.log
}

counter=$(check_nginx)  # 主逻辑
if [ "${counter}" -eq 0 ]; then
    log_message "$NGINX_SERVICE is not running, attempting to start..."
    systemctl start $NGINX_SERVICE
    sleep $RETRY_DELAY
    counter=$(check_nginx)
    if [ "${counter}" -eq 0 ]; then
        log_message "$NGINX_SERVICE failed to start, stopping $KEEPALIVED_SERVICE..."
        systemctl stop $KEEPALIVED_SERVICE
    else
        log_message "$NGINX_SERVICE started successfully."
    fi
else
    log_message "$NGINX_SERVICE is running."
fi
# 防止日志过大启动日志轮转
[root@LbA ~]# vim /etc/logrotate.d/nginx_keepalived_check
/var/log/nginx_keepalived_check.log {
    daily
    rotate 1
    missingok
    notifempty
    compress
    delaycompress
    postrotate
        systemctl reload nginx > /dev/null 2>&1 || true
    endscript
}
# 配置Keepalived自启动
[root@LbA ~]# systemctl enable --now keepalived

LbB安装并配置Keepalived

除了keepalived.conf不一样其他步骤都一样，LbB的keepalived.conf如下

[root@LbB ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LBB_BACKUP	# 不同点
}

vrrp_script chk_nginx {
    script "/usr/local/bin/check_nginx.sh"
    interval 2
    weight -5
}

vrrp_instance VI_1 {
    state BACKUP	# 不同点
    interface ens160
    virtual_router_id 51
    priority 100	# 不同点
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.100.99/24
    }
    track_script {
        chk_nginx
    }
}

安装并配置Nginx（两节点配置一致）

# 添加最新版Nginx仓库源
[root@LbA ~]# vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
# 安装Nginx和四层代理模块
[root@LbA ~]# yum install nginx nginx-mod-stream -y
# 创建四层代理配置路径
[root@LbA ~]# mkdir -p /etc/nginx/stream.d
# 配置四层代理
[root@LbA ~]# vim /etc/nginx/nginx.conf
# 这是主配置文件不进行任何修改最后加上四层代理块，注意要和http块同级
...
http {
...
}
stream {
    include /etc/nginx/stream.d/*.conf;	# 四层代理配置文件位置
}
# 四层代理配置
[root@LbA ~]# vim /etc/nginx/stream.d/k8s-api.conf
# 三个控制平面
upstream k8s_api {
    server masterA.k8s.local:6443;
    server masterB.k8s.local:6443;
    server masterC.k8s.local:6443;
}

server {
    listen 6443;	# 监听端口
    proxy_pass k8s_api;
}
# 自启动Nginx
[root@LbA ~]# systemctl enable --now nginx

大家可以更改nginx配置文件把他改错停止服务并无法启动，查看keepalived服务是否配置正常，VIP是否正常漂移到LbB节点

现在把kubernete三个控制平面的hosts中的负载均衡器改成keepalived的VIP验证集群是否正常启动

[root@masterA ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.99 control-plane-endpoint.k8s.local control-plane-endpoint	# 修改此IP
192.168.100.10 masterA.k8s.local masterA
192.168.100.20 masterB.k8s.local masterB
192.168.100.30 masterC.k8s.local masterC
192.168.100.11 workA.k8s.local workA
192.168.100.22 workB.k8s.local workB
192.168.100.33 workC.k8s.local workC
# 查看集群节点，可以看见可以正常获取到集群正常，状态为“NotReady”是因为我们网络组件还没有安装
[root@masterA ~]# kubectl get nodes
NAME                STATUS     ROLES           AGE    VERSION
mastera.k8s.local   NotReady   control-plane   2d1h   v1.31.0
masterb.k8s.local   NotReady   control-plane   2d1h   v1.31.0
masterc.k8s.local   NotReady   control-plane   2d1h   v1.31.0
worka.k8s.local     NotReady   <none>          2d     v1.31.0
workb.k8s.local     NotReady   <none>          2d     v1.31.0
workc.k8s.local     NotReady   <none>          2d     v1.31.0

安装网络组件（单独讲解）