k8s svc通过externalIPs并结合Keepalived对外暴露服务

最新推荐文章于 2024-06-04 20:56:10 发布

Hlroliu

最新推荐文章于 2024-06-04 20:56:10 发布

阅读量2.7k

点赞数 1

分类专栏： kubernetes 文章标签： kubernetes 运维

本文链接：https://blog.csdn.net/Hlroliu/article/details/128548550

版权

kubernetes 专栏收录该内容

7 篇文章 1 订阅

订阅专栏

Kubemetes 的 Service需要对集群外暴露，可以通过NodePort、Ingress和LoadBalancer，其中
NodePort：需要预先知道k8s集群节点的IP地址，在使用时需指定node的ip（这样配置存在单点故障）；
Ingress：实现的是HTTP(S)负载均衡器，只能代理七层；
LoadBalancer：需要通过云服务商提供的负载均衡器将服务暴露到集群外部。
在非公有云环境的k8s集群上，ClusterIp类型的Service可通过externalIPs设置一个外部的 IP 地址，并且将流量导入到集群内部。externalIps(外部IP)要求是至少能路由到一个k8s节点上。即如果有外部IP可以路由到一个或多个k8s节点上，就可以把k8s的Service暴露在这个外部IP上，通过访问外部IP+Service的端口将流量接入到集群内。这个IP再通过Keepalived配置为VIP，浮动于多个node节点上，即可避免NodePort单点故障问题。以下是实现步骤

一、基础环境

一个自建的k8s集群，测试环境配置如下

kubectl get node -owide
NAME            STATUS                     ROLES    AGE     VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
192.168.5.124   Ready,SchedulingDisabled   master   3h13m   v1.21.12   192.168.5.124   <none>        Ubuntu 20.04.3 LTS   5.4.0-121-generic   docker://19.3.12
192.168.5.134   Ready                      node     3h11m   v1.21.12   192.168.5.134   <none>        Ubuntu 20.04.3 LTS   5.4.0-81-generic    docker://19.3.12
192.168.5.144   Ready                      node     3h11m   v1.21.12   192.168.5.144   <none>        Ubuntu 20.04.3 LTS   5.4.0-81-generic    docker://19.3.12
master节点网卡信息
ifconfig
ens32: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.124  netmask 255.255.255.0  broadcast 192.168.5.255
node1节点网卡信息
ifconfig
ens32: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.134  netmask 255.255.255.0  broadcast 192.168.5.255
node2节点网卡信息
ens32: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.144  netmask 255.255.255.0  broadcast 192.168.5.255

二、Keepalived生成externalIPs

在node1 和 node2上安装配置Keepalived
node1 Keepalived配置

cat /etc/keepalived/keepalived.conf 
global_defs {
    notification_email {
        root@localhost 
    }
    notification_email_from keepalived@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id node1
    vrrp_skip_check_adv_addr 
    #vrrp_strict 
    vrrp_garp_interval 0 
    vrrp_gna_interval 0 
    vrrp_mcast_group4 224.0.0.18 
}
vrrp_instance EIP {
    state MASTER 
    interface ens32
    virtual_router_id 66 
    priority 100 
    advert_int 1
    authentication {
        auth_type PASS #预共享密钥认证，同一个虚拟路由器的keepalived节点必须一样
        auth_pass 12345678
    }
    virtual_ipaddress {
        192.168.5.200 dev ens32 label ens32:0
    }
}

node2 Keepalived配置

cat /etc/keepalived/keepalived.conf 
global_defs {
    notification_email {
        root@localhost 
    }
    notification_email_from keepalived@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id node2
    vrrp_skip_check_adv_addr 
    #vrrp_strict
    vrrp_garp_interval 0 
    vrrp_gna_interval 0
    vrrp_mcast_group4 224.0.0.18
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    virtual_router_id 66 
    priority 80 
    advert_int 1
    authentication {
        auth_type PASS #预共享密钥认证，同一个虚拟路由器的keepalived节点必须一样
        auth_pass 12345678
    }
    virtual_ipaddress {
        192.168.5.200 dev ens32 label ens32:0
    }
}

启动Keepalived并验证

systemctl daemon-reload
systemctl start keepalived.service
systemctl enable keepalived.service
systemctl status keepalived.service 
● keepalived.service - Keepalive Daemon (LVS and VRRP)
     Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-01-04 14:26:40 CST; 1h 12min ago
   Main PID: 156590 (keepalived)
      Tasks: 2 (limit: 9413)
     Memory: 1.8M
     CGroup: /system.slice/keepalived.service
             ├─156590 /usr/sbin/keepalived --dont-fork
             └─156603 /usr/sbin/keepalived --dont-fork

查看VIP信息

root@node1:~# ifconfig ens32:0
ens32:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.200  netmask 255.255.255.255  broadcast 0.0.0.0
root@node2:~# ping -c4 192.168.5.200
PING 192.168.5.200 (192.168.5.200) 56(84) bytes of data.
64 bytes from 192.168.5.200: icmp_seq=1 ttl=64 time=0.061 ms
64 bytes from 192.168.5.200: icmp_seq=2 ttl=64 time=0.074 ms
64 bytes from 192.168.5.200: icmp_seq=3 ttl=64 time=0.071 ms
64 bytes from 192.168.5.200: icmp_seq=4 ttl=64 time=0.068 ms

--- 192.168.5.200 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3060ms
rtt min/avg/max/mdev = 0.061/0.068/0.074/0.004 ms

三、创建svc并指定externalIPs

借用大佬的镜像创建一个deployment，docker hub地址：https://hub.docker.com/r/ikubernetes/demoapp

cat nginx.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deploy
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: ikubernetes/demoapp:v1.0 
---
apiVersion: v1
kind: Service
metadata:
  labels:       
    app: nginx-svc         
  name: nginx-svc          
  namespace: default        
spec:
  ports:                
  - name: http   
    port: 80             
    targetPort: 80        
    protocol: TCP               
  selector:
    app: nginx
  externalIPs:
  - 192.168.5.200	# 设置 externalIPs 为VIP
创建资源
kubectl apply -f nginx.yaml
kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
nginx-deploy-7788867569-hdm4g   1/1     Running   0          64m
nginx-deploy-7788867569-wcqhp   1/1     Running   0          64m
kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP     PORT(S)   AGE
nginx-svc    ClusterIP   10.96.225.101   192.168.5.200   80/TCP    70m

访问测试

while true; do curl 192.168.5.200;sleep 1; done
iKubernetes demoapp v1.0 !! ClientIP: 10.244.104.0, ServerName: nginx-deploy-7788867569-wcqhp, ServerIP: 10.244.166.134!
iKubernetes demoapp v1.0 !! ClientIP: 192.168.5.144, ServerName: nginx-deploy-7788867569-hdm4g, ServerIP: 10.244.104.5!
iKubernetes demoapp v1.0 !! ClientIP: 10.244.104.0, ServerName: nginx-deploy-7788867569-wcqhp, ServerIP: 10.244.166.134!
iKubernetes demoapp v1.0 !! ClientIP: 192.168.5.144, ServerName: nginx-deploy-7788867569-hdm4g, ServerIP: 10.244.104.5!
iKubernetes demoapp v1.0 !! ClientIP: 10.244.104.0, ServerName: nginx-deploy-7788867569-wcqhp, ServerIP: 10.244.166.134!
iKubernetes demoapp v1.0 !! ClientIP: 192.168.5.144, ServerName: nginx-deploy-7788867569-hdm4g, ServerIP: 10.244.104.5!
iKubernetes demoapp v1.0 !! ClientIP: 10.244.104.0, ServerName: nginx-deploy-7788867569-wcqhp, ServerIP: 10.244.166.134!
iKubernetes demoapp v1.0 !! ClientIP: 192.168.5.144, ServerName: nginx-deploy-7788867569-hdm4g, ServerIP: 10.244.104.5!
iKubernetes demoapp v1.0 !! ClientIP: 10.244.104.0, ServerName: nginx-deploy-7788867569-wcqhp, ServerIP: 10.244.166.134!

保持以上访问，停止node1上的Keepalived，模拟节点故障

root@node1:~# systemctl stop keepalived.service 
root@node1:~# ifconfig ens32:0
ens32:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
IP漂移至node2节点，curl访问也未出现故障
root@node2:~# ifconfig ens32:0
ens32:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.200  netmask 255.255.255.255  broadcast 0.0.0.0