继续上一章部署。
七、部署高可用组件
本章介绍keepalived和haproxy实现kube-apiserver高可用。
- keepalive 提供 kube-apiserver 对外提供服务的VIP;
- haproxy 监听VIP,后端链接所有 kube-apiserver 实例,提供健康检查和负载均衡功能
- keepalived 一主一备的运行模式,本文档复用 master 节点的两台设备
- haproxy 监听 8443 端口,与 kube-apiserver 的 6443 端口区分开
- keepalived 在运行过程中周期检查本机的 haproxy 进程状态,如果检测到 haproxy 进程异常,则触发重新选主的过程,将VIP飘逸到新选出来的主节点,从而实现 VIP 的高可用
- 所有组件都通过 VIP 和 haproxy 监听的 8443 端口访问 kube-apiserver
7.1 安装软件包
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh k8s@${master_ip} "sudo yum install -y keepalived haproxy" done
7.2 配置 haproxy
cat > haproxy.cfg <<EOF global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /var/run/haproxy-admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon nbproc 1 defaults log global timeout connect 5000 timeout client 10m timeout server 10m listen admin_stats bind 0.0.0.0:10080 mode http log 127.0.0.1 local0 err stats refresh 30s stats uri /status stats realm welcome login\ Haproxy stats auth admin:123456 stats hide-version stats admin if TRUE listen kube-master bind 0.0.0.0:8443 mode tcp option tcplog balance source server 192.168.56.20 192.168.56.20:6443 check inter 2000 fall 2 rise 2 weight 1 server 192.168.56.21 192.168.56.21:6443 check inter 2000 fall 2 rise 2 weight 1 EOF
- haproxy 在10080端口输出 status 信息
- haproxy 监听所有接口的 8443 端口,该端口与环境变量 ${KUBE_APISERVER} 指定的端口必须一致
- server 字段列出所有 kube-apiserver 监听的 IP 和端口
7.3 下发 haproxy.cfg 到所有 master 节点
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" scp haproxy.cfg root@${master_ip}:/etc/haproxy done
7.4 起 haproxy 服务
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh k8s@${master_ip} "sudo systemctl restart haproxy" done
7.5 检查服务状态
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh k8s@${master_ip} "sudo systemctl status haproxy|grep Active" done
确保状态都是 active (running)。
查看日志:journalctl -u haproxy
7.6 检查是否监听了 8443 端口
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh k8s@${master_ip} "sudo netstat -lnpt|grep haproxy" done
7.7 配置 keeplived
keepalived 是一主一备的运行模式,所以配置文件类型有两种
master 192.168.56.20 backup 192.168.56.21
master配置文件:
source /opt/k8s/bin/environment.sh cat > keepalived-master.conf <<EOF global_defs { router_id lb-master-105 } vrrp_script check-haproxy { script "killall -0 haproxy" interval 5 weight -30 } vrrp_instance VI-kube-master { state MASTER priority 120 dont_track_primary interface ${VIP_IF} virtual_router_id 68 advert_int 3 track_script { check-haproxy } virtual_ipaddress { ${MASTER_VIP} } } EOF
- 两个地方需要注意 ${VIP_IF} 和 ${MASTER_VIP}
- 使用 killall -0 haproxy 命令检查所在节点的 haproxy 进程是否正常。如果异常则将权重减少(-30),从而触发重新选主的过程
- route_id、virtual_router_id 用于标识属于该 HA 的keepalived 实例,如果有多套keepalived HA,则必须各不相同
backup 配置文件:
source /opt/k8s/bin/environment.sh cat > keepalived-backup.conf <<EOF global_defs { router_id lb-backup-105 } vrrp_script check-haproxy { script "killall -0 haproxy" interval 5 weight -30 } vrrp_instance VI-kube-master { state BACKUP priority 110 dont_track_primary interface ${VIP_IF} virtual_router_id 68 advert_int 3 track_script { check-haproxy } virtual_ipaddress { ${MASTER_VIP} } } EOF
- priority 的值必须小于 master
7.8 下发keepalived配置文件
[root@k8s-m1 ha_config]# scp keepalived-master.conf root@192.168.56.20:/etc/keepalived/keepalived.conf [root@k8s-m1 ha_config]# scp keepalived-backup.conf root@192.168.56.21:/etc/keepalived/keepalived.conf
7.9 起 keepalived 服务
source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh root@${master_ip} "systemctl restart keepalived" done
7.10 检查服务状态
# 确认服务是 active (running) source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh root@${master_ip} "systemctl status keepalived|grep Active" done # 查看 VIP 所在的节点,确保可以 ping 通 VIP source /opt/k8s/bin/environment.sh for master_ip in ${MASTER_IPS[@]} do echo ">>> ${master_ip}" ssh root@${master_ip} "/usr/sbin/ip addr show ${VIP_IF}" ssh root@${master_ip} "ping -c 1 ${MASTER_VIP}" done # 查看日志 journalctl -u keepalived
7.11 浏览器打开status也没
地址:${MASTER_VIP}:10080/status
用户名和密码在 haproxy.cfg 的 status auth 字段
HAProxy status页面