HAproxy+keepalived instance of production

QKCP LB 安装：高可用 Kubernetes 集群能够确保应用程序在运行时不会出现服务中断。

• 正式部署由客户提供资源，客户自行负责客户数据；
• LB资源需检查及验证；
• 测试环境，可基于虚机搭建，提供存储资源。

集群架构图：

示例集群有三个主节点，三个工作节点，两个用于负载均衡的节点，以及一个虚拟 IP 地址。本示例中的虚拟 IP 地址也可称为“浮动 IP 地址”。这意味着在节点故障的情况下，该 IP 地址可在节点之间漂移，从而实现高可用。

请注意，在本示例中，Keepalived 和 HAproxy 安装在单独的两台节点上。也可以，单独配置两个用于负载均衡的特定节点（如上图，您可以按需增加更多此类节点）会更加安全、灵活。

 

配置负载均衡

Keepalived 提供 VRPP 实现，并允许您配置 Linux 机器使负载均衡，预防单点故障。HAProxy 提供可靠、高性能的负载均衡，能与 Keepalived 完美配合。

由于 lb1 和 lb2 上安装了 Keepalived 和 HAproxy，如果其中一个节点故障，虚拟 IP 地址（即浮动 IP 地址）将自动与另一个节点关联，使集群仍然可以正常运行，从而实现高可用。若有需要，也可以此为目的，添加更多安装 Keepalived 和 HAproxy 的节点。

安装Keepalived 和HAproxy

ha01与ha02节点通过apt安装HAProxy和KeepAlived：

apt install keepalived haproxy -y

ha01与ha02节点配置HAProxy（ha1与ha2节点的HAProxy配置相同）：

vim /etc/haproxy/haproxy.cfg

HAproxy配置文件：(更多注解见source code下的/doc/intro.txt和/doc/configuration.txt)

root@prd-lb01:~# cat /etc/haproxy/haproxy.cfg

global

        maxconn  2000

        ulimit-n  16384

        log  127.0.0.1 local0 err

        stats timeout 30s

defaults

        log global

        mode  http    #工作在layer7，Haproxy最具有价值的部分

        option  httplog

        timeout connect 5000

        timeout client  50000

        timeout server  50000

        timeout http-request 15s

        timeout http-keep-alive 15s

frontend monitor-in

        bind *:33305

        mode http  

        option httplog

        monitor-uri /monitor

frontend k8s-master

        bind 0.0.0.0:16443

        bind 127.0.0.1:16443

        mode tcp     #工作在layer4，也是缺省配置，四层负载

        option tcplog

        tcp-request inspect-delay 5s

        default_backend qyk8s-master   #定位到后端qyk8s-master

backend qyk8s-master

        mode tcp   #layer4必须和前端k8s-master对应

        option tcplog

        option tcp-check

        balance roundrobin  #负载的算法

        default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100

        server host-master01 10.0.200.208:6443  check

        server host-master02 10.0.200.209:6443  check

        server host-master03 10.0.200.210:6443  check

KeepAlived配置文件

ha1与ha2节点配置KeepAlived，配置不一样

ha1节点的配置：

mkdir /etc/keepalived

vim /etc/keepalived/keepalived.conf

 

root@prd-lb01:~# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {       router_id LVS_DEVEL } vrrp_script chk_apiserver {         script "/etc/keepalived/check_apiserver.sh"         interval 5         weight -5         fall 2 rise 1 } vrrp_instance VI_1 {        state MASTER   #master角色        interface ens3    #网卡名称为ens3       # mcast_src_ip 10.0.200.217 #发送多播包的地址，如果不设置默认使用绑定网卡的primary ip        virtual_router_id 151    #VRID标记,路由ID，可通过#tcpdump vrrp查看        priority 101                   #优先级，高优先级竞选为master        advert_int 2        authentication {         auth_type PASS            #认证类型         auth_pass K8SHA_KA_AUTH   #认证的口令     }         virtual_ipaddress {               10.0.200.221/24                                      #设置vip     }         track_script {         chk_apiserver     } }

 

ha2节点的配置：

root@prd-lb02:~# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {     router_id LVS_DEVEL } vrrp_script chk_apiserver {     script "/etc/keepalived/check_apiserver.sh"    interval 5     weight -5     fall 2 rise 1 } vrrp_instance VI_1 {     state BACKUP   #设置为backup角色     interface ens3     #mcast_src_ip 10.0.200.218     virtual_router_id 151     priority 100    #优先级低于master的101     advert_int 2     authentication {         auth_type PASS         auth_pass K8SHA_KA_AUTH     }     virtual_ipaddress {         10.0.200.221/24     }     track_script {        chk_apiserver     } }

#    track_script { #       chk_apiserver #    }

配置KeepAlived健康检查文件：

cat /etc/keepalived/check_apiserver.sh

#!/bin/bash err=0 for k in $(seq 1 3) do     check_code=$(pgrep haproxy)     if [[ $check_code == "" ]]; then         err=$(expr $err + 1)         sleep 1         continue     else         err=0         break     fi done if [[ $err != "0" ]]; then     echo "systemctl stop keepalived"     /usr/bin/systemctl stop keepalived     exit 1 else     exit 0 fi

添加脚本执行权限：

chmod +x /etc/keepalived/check_apiserver.sh

启动haproxy和keepalived

systemctl restart haproxy

systemctl restart keepalived

systemctl enable haproxy && systemctl enable keepalived

HAProxy和KeepAlived测试

重要：安装keepalived和haproxy后，需要测试keepalived是否是正常的

root@prd-master01:~# telnet 10.0.200.220 16443 Trying 10.0.200.220... Connected to 10.0.200.220. Escape character is '^]'.

如果telnet没有出现'^]' ，则认为VIP不可用，不可在继续往下执行，需要排查keepalived的问题，比如防火墙和selinux，haproxy和keepalived的状态，监听端口等。

 

北向流星转发Nginx

公网映射如下表可查：

<<nat_imexport_1692261237724.xlsx>>

公网的流量访问是通过Firewall的SNAT进入，配置了集群外的LB，使用两台nginx做负载流量转发。

[root@k8s-ng2 conf]# cat nginx.conf

worker_processes  1;

events {

    worker_connections  1024;

}

http {

    include       mime.types;

    #核心模块指令默认设置为二进制流，当文件类型未定义时使用这种方式

    default_type  application/octet-stream; 

        #sendfile 指令指定 nginx 是否调用 sendfile 函数（zero copy 方式）来输出文件，对于普通应用必须设为 on,

        #如果用来进行下载等应用磁盘IO重负载应用，可设置为 off，以平衡磁盘与网络I/O处理速度，降低系统的uptime.

    sendfile        on;

    keepalive_timeout  65;  #客户端长连接保存活动的超时时间，单位是秒

    server {

        listen       80;

        server_name  localhost;

        location / {

            root   html;

            index  index.html index.htm;

        }

        error_page   500 502 503 504  /50x.html;

        location = /50x.html {

            root   html;

        }

    }

    server {

        listen       8081 ssl;   //监听8081端口，即ssl

        server_name  localhost;

        ssl_certificate      /data/cert/8631154__holitech.net.pem; //指定pem文件路径

        ssl_certificate_key  /data/cert/8631154__holitech.net.key; //指定key文件路径

        ssl_session_cache    shared:SSL:1m;  //指定session cache大小

        ssl_session_timeout  5m;          //指定session超时时间

        ssl_ciphers  HIGH:!aNULL:!MD5;   //指定ssl算法

        ssl_prefer_server_ciphers  on;  //优先采取服务器算法

        client_max_body_size 500M;

     #对 "/" 启用反向代理

        location / {

            proxy_pass https://srmprd.holitech.net:30341/; #http代理，使用dns后台轮询

            proxy_http_version 1.1;  //表示反向代理发送的HTTP协议的版本是1.1，HTTP1.1支持长连接

    #配置支持webSocket开始

            proxy_set_header Upgrade $http_upgrade;

            proxy_set_header Connection "Upgrade";

    #配置支持webSocket结束

        }

    }

     server {

        listen       8082 ssl;

        server_name  localhost;

        ssl_certificate      /data/cert/8631154__holitech.net.pem;

        ssl_certificate_key  /data/cert/8631154__holitech.net.key;

        ssl_session_cache    shared:SSL:1m;

        ssl_session_timeout  5m;

        ssl_ciphers  HIGH:!aNULL:!MD5;

        ssl_prefer_server_ciphers  on;

        client_max_body_size 500M;

        location / {

            proxy_pass http://10.0.200.210:31437;   #http代理，使用指定IP访问

            proxy_http_version 1.1;

            proxy_set_header Upgrade $http_upgrade;

            proxy_set_header Connection "Upgrade";

        }

    }

}

----------------------------------------------------------------

[root@k8s-ng2 conf]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

   router_id lb224

}

vrrp_instance paas {

    state MASTER

    interface eth0

    virtual_router_id 88

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.0.200.226

    }

}

----------------------------------------------------------------

[root@k8s-ng1 ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

   router_id lb223

}

vrrp_instance paas {

    state BACKUP

    interface eth0

    virtual_router_id 88

    priority 50

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.0.200.226

    }

}

---------------------------------------------------------------

[root@k8s-ng1 ~]# cat /usr/local/nginx/conf/nginx.conf

worker_processes  1;

events {

    worker_connections  1024;

}

http {

    include       mime.types;

    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;

    server {

        listen       80;

        server_name  localhost;

        location / {

            root   html;

            index  index.html index.htm;

        }

        error_page   500 502 503 504  /50x.html;

        location = /50x.html {

            root   html;

        }

    }

    server {

        listen       8081 ssl;

        server_name  localhost;

        ssl_certificate      /data/cert/8631154__holitech.net.pem;

        ssl_certificate_key  /data/cert/8631154__holitech.net.key;

        ssl_session_cache    shared:SSL:1m;

        ssl_session_timeout  5m;

        ssl_ciphers  HIGH:!aNULL:!MD5;

        ssl_prefer_server_ciphers  on;

        client_max_body_size 500M;

        location / {

            proxy_pass https://srmprd.holitech.net:30341;  #

            proxy_http_version 1.1;

            proxy_set_header Upgrade $http_upgrade;

            proxy_set_header Connection "Upgrade";

        }

    }

    server {

        listen       8082 ssl;

        server_name  localhost;

        ssl_certificate      /data/cert/8631154__holitech.net.pem;

        ssl_certificate_key  /data/cert/8631154__holitech.net.key;

        ssl_session_cache    shared:SSL:1m;

        ssl_session_timeout  5m;

        ssl_ciphers  HIGH:!aNULL:!MD5;

        ssl_prefer_server_ciphers  on;

        client_max_body_size 500M;

        location / {

            proxy_pass http://10.0.200.210:31437;     #

        }

    }  

}

 

root@host-lb01:~# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {       router_id LVS_DEVEL } vrrp_script chk_apiserver {         script "/etc/keepalived/check_apiserver.sh"         interval 5         weight -5         fall 2  rise 1 } vrrp_instance VI_1 {        state MASTER        interface ens3       # mcast_src_ip 10.0.200.206 #发送多播包的地址，如果不设置默认使用绑定网卡的primary ip        virtual_router_id 150    #VRID标记 ,路由ID，可通过#tcpdump vrrp查看        priority 101                   #优先级，高优先级竞选为master        advert_int 2        authentication {         auth_type PASS         auth_pass K8SHA_KA_AUTH     }         virtual_ipaddress {               10.0.200.220/24                                      #设置vip     }         track_script {         chk_apiserver     } }

Host-lb01的keepalived.conf

root@host-lb01:~# cat /etc/haproxy/haproxy.cfg global         maxconn  2000         ulimit-n  16384         log  127.0.0.1 local0 err         stats timeout 30s defaults         log global         mode  http         option  httplog         timeout connect 5000         timeout client  50000         timeout server  50000         timeout http-request 15s         timeout http-keep-alive 15s frontend monitor-in         bind *:33305         mode http         option httplog         monitor-uri /monitor frontend ks-console         bind 0.0.0.0:80         bind 127.0.0.1:80         mode http         option httplog         default_backend qyks-console frontend k8s-master         bind 0.0.0.0:16443         bind 127.0.0.1:16443         mode tcp         option tcplog         tcp-request inspect-delay 5s         default_backend qyk8s-master backend qyk8s-master         mode tcp         option tcplog         option tcp-check         balance roundrobin         default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100         server host-master01 10.0.200.201:6443  check         server host-master02 10.0.200.202:6443  check         server host-master03 10.0.200.203:6443  check backend qyks-console         mode http         option httplog         option tcp-check         balance roundrobin         default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100         server host-master01 10.0.200.201:30880  check         server host-master02 10.0.200.202:30880  check         server host-master03 10.0.200.203:30880  check

Host-lb01的Haproxy.cfg

ubuntu@host-lb02:~$ cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs {     router_id LVS_DEVEL } vrrp_script chk_apiserver {     script "/etc/keepalived/check_apiserver.sh"    interval 5     weight -5     fall 2  rise 1 } vrrp_instance VI_1 {     state BACKUP     interface ens3     #mcast_src_ip 10.236.2.34     virtual_router_id 150     priority 100     advert_int 2     authentication {         auth_type PASS         auth_pass K8SHA_KA_AUTH     }     virtual_ipaddress {         10.0.200.220/24     }     track_script {        chk_apiserver     } }

host-lb02的Keepalived.conf

ubuntu@host-lb02:~$ cat /etc/haproxy/haproxy.cfg global         maxconn  2000         ulimit-n  16384         log  127.0.0.1 local0 err         stats timeout 30s defaults         log global         mode  http         option  httplog         timeout connect 5000         timeout client  50000         timeout server  50000         timeout http-request 15s         timeout http-keep-alive 15s frontend monitor-in         bind *:33305         mode http         option httplog         monitor-uri /monitor frontend ks-console         bind 0.0.0.0:80         bind 127.0.0.1:80         mode http         option httplog         default_backend qyks-console frontend k8s-master         bind 0.0.0.0:16443         bind 127.0.0.1:16443         mode tcp         option tcplog         tcp-request inspect-delay 5s         default_backend qyk8s-master backend qyk8s-master         mode tcp         option tcplog         option tcp-check         balance roundrobin         default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100         server host-master01 10.0.200.201:6443  check         server host-master02 10.0.200.202:6443  check         server host-master03 10.0.200.203:6443  check backend qyks-console         mode http         option httplog         option tcp-check         balance roundrobin         default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100         server host-master01 10.0.200.201:30880  check         server host-master02 10.0.200.202:30880  check         server host-master03 10.0.200.203:30880  check

 

 

ps，实际项目的东西，为了安全，就贴一个图把