K8s 中 iptables 和ipvs 的理解

Iptables

定义service 类型为 ClusterIP

kubectl get svc -owide
nginx-basic   ClusterIP   10.101.157.173   <none>        80/TCP    42h   app=nginx

kubectl get pods -owide

nginx-deployment-6799fc88d8-dmhk9   1/1     Running   0          27m   10.224.0.11   node1   <none>           <none>
nginx-deployment-6799fc88d8-m6vsc   1/1     Running   0          27m   10.224.0.9    node1   <none>           <none>
nginx-deployment-6799fc88d8-mmgxl   1/1     Running   0          27m   10.224.0.10   node1   <none>           <none>

# 使用iptables-save 查看路由规则

iptables-save -t nat

-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A add -m message -j jump  所有进来的包在出prerouting 这个包都要去kube-service 中去看看

-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A add -m message -j jump  所有出来的包在出prerouting 这个包都要去kube-service 中去看看

-A KUBE-SERVICES -d 10.101.157.173/32 -p tcp -m comment --comment "default/nginx-basic:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-WWRFY3PZ7W3FGMQW

-d destination(目标)

iptable 规则说明一条一套执行  有百分之33的几率被KUBE-SEP-ZMFA63VDZWTWAEBQ 这个chain 处理
-A KUBE-SVC-WWRFY3PZ7W3FGMQW -m comment --comment "default/nginx-basic:http" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-ZMFA63VDZWTWAEBQ
      
    -A KUBE-SEP-ZMFA63VDZWTWAEBQ -s 10.224.0.10/32 -m comment --comment "default/nginx-basic:http" -j KUBE-MARK-MASQ
    -A KUBE-SEP-ZMFA63VDZWTWAEBQ -p tcp -m comment --comment "default/nginx-basic:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination

-A KUBE-SVC-WWRFY3PZ7W3FGMQW -m comment --comment "default/nginx-basic:http" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-B7KA7M3EBT33ZNQX

    -A KUBE-SEP-B7KA7M3EBT33ZNQX -s 10.224.0.11/32 -m comment --comment "default/nginx-basic:http" -j KUBE-MARK-MASQ
    -A KUBE-SEP-B7KA7M3EBT33ZNQX -p tcp -m comment --comment "default/nginx-basic:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination


-A KUBE-SVC-WWRFY3PZ7W3FGMQW -m comment --comment "default/nginx-basic:http" -j KUBE-SEP-YUJMBOAQBKLEZT3R

    -A KUBE-SEP-YUJMBOAQBKLEZT3R -s 10.224.0.9/32 -m comment --comment "default/nginx-basic:http" -j KUBE-MARK-MASQ 
    -A KUBE-SEP-YUJMBOAQBKLEZT3R -p tcp -m comment --comment "default/nginx-basic:http" -m tcp -j DNAT --to-destination :0 --persistent --to-destination :0 --persistent --to-destination

最终结论为：从上可以看出k8s 通过service 的流量会路由到pod 中去 

定义service 类型为 NodePort

-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-basic:http" -m tcp --dport 31219 -j KUBE-SVC-WWRFY3PZ7W3FGMQW
 netstat -na|grep 31219
tcp        0      0 0.0.0.0:31219           0.0.0.0:*               LISTEN
 netstat -nap|grep 31219
tcp        0      0 0.0.0.0:31219           0.0.0.0:*               LISTEN      52480/kube-proxy

查看-j KUBE-SVC-WWRFY3PZ7W3FGMQW 与上图service类型为ClusterIP一致 也是往三个Pod 中转发流量

ipvs

IPVS 
  修改kube-proxy 的configmap中修改对应的mode 然后改为ipvs

 ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  node1:31219 rr
  -> 10.224.0.9:http              Masq    1      0          0
  -> 10.224.0.10:http             Masq    1      0          0
  -> 10.224.0.11:http             Masq    1      0          0
TCP  node1:31219 rr
  -> 10.224.0.9:http              Masq    1      0          0
  -> 10.224.0.10:http             Masq    1      0          0
  -> 10.224.0.11:http             Masq    1      0          0
TCP  node1:https rr
  -> node1:sun-sr-https           Masq    1      4          0
TCP  node1:domain rr
  -> 10.224.0.7:domain            Masq    1      0          0
  -> 10.224.0.8:domain            Masq    1      0          0
TCP  node1:9153 rr
  -> 10.224.0.7:9153              Masq    1      0          0
  -> 10.224.0.8:9153              Masq    1      0          0
TCP  node1:http rr
  -> 10.224.0.9:http              Masq    1      0          0
  -> 10.224.0.10:http             Masq    1      0          0
  -> 10.224.0.11:http             Masq    1      0          0
TCP  node1:31219 rr
  -> 10.224.0.9:http              Masq    1      0          0
  -> 10.224.0.10:http             Masq    1      0          0
  -> 10.224.0.11:http             Masq    1      0          0
TCP  node1:31219 rr
  -> 10.224.0.9:http              Masq    1      0          0
  -> 10.224.0.10:http             Masq    1      0          0
  -> 10.224.0.11:http             Masq    1      0          0
UDP  node1:domain rr
  -> 10.224.0.7:domain            Masq    1      0          0
  -> 10.224.0.8:domain            Masq    1      0          0

  使用iptables -F -t nat
  然后再次查看下面的规则 会发现多了一条IPSET 现在使用ipvs 的话会做一个 IP 伪装 KUBE-MARK-MASQ
  iptables-save -t nat
# Generated by iptables-save v1.4.21 on Wed Feb 16 13:31:44 2022
*nat
:PREROUTING ACCEPT [32:1440]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [47:2820]
:POSTROUTING ACCEPT [47:2820]
:DOCKER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SEP-B7KA7M3EBT33ZNQX - [0:0]
:KUBE-SEP-DWSBRBKMJKQCC6K6 - [0:0]
:KUBE-SEP-H7GRETGDBNJH3TNF - [0:0]
:KUBE-SEP-M7D4TRPAWXXY3RIC - [0:0]
:KUBE-SEP-OZ2EEVAFCTBCWHHW - [0:0]
:KUBE-SEP-P6X6TGMQCZ3YXVTJ - [0:0]
:KUBE-SEP-TGTH74NZUASQJRDW - [0:0]
:KUBE-SEP-WQDGZCN3UF6UPQZM - [0:0]
:KUBE-SEP-YUJMBOAQBKLEZT3R - [0:0]
:KUBE-SEP-ZMFA63VDZWTWAEBQ - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-JD5MR3NA4I4DYORP - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:KUBE-SVC-WWRFY3PZ7W3FGMQW - [0:0]
:cali-OUTPUT - [0:0]
:cali-POSTROUTING - [0:0]
:cali-PREROUTING - [0:0]
:cali-fip-dnat - [0:0]
:cali-fip-snat - [0:0]
:cali-nat-outgoing - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE
-A KUBE-SERVICES ! -s 10.244.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
COMMIT