Kubernetes 开启iptables后 规则添加以及验证

K8s开启iptables后规则设置和验证

1.Master node需要配置的iptables规则

1.1 INPUT链

iptables的表，链，端口对应的服务地址

Protocol	iptables table	ipables chain	Port Range	Purpose	Used by
UDP	filter	INPUT	8472	flannel	network
TCP	filter	INPUT	6443	Kubernetes API server	ALL node
TCP	filter	INPUT	2379-2380	etcd server client API	kube-apiserver,etcd
TCP	filter	INPUT	10250	kubelet API	Control plane, Self ,kubectl exec
TCP	filter	INPUT	10251	kube-scheduler	self
TCP	filter	INPUT	10252	kube-controller-manager	self

1.1.1 pod通讯通过flannel的udp端口8472通讯

1.1.1.1包封装图

1.1.1.2不通的情况

启动node2的iptables，确定8472udp端口不通的情况，在node1的pod上测试node2上的pod是否能ping通

# kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
busybox                     1/1     Running   0          162m    10.244.1.48   xen11-196   <none>           <none>
my-nginx-75897978cd-64zj2   1/1     Running   0          4h15m   10.244.1.39   xen11-196   <none>           <none>
my-nginx-75897978cd-bbrtm   1/1     Running   0          23d     10.244.1.29   xen11-196   <none>           <none>
my-nginx-75897978cd-t7s6n   1/1     Running   0          142m    10.244.2.2    xen11-197   <none>           <none>

Pod IP列表

pod	node
10.244.2.2	xen11-197
10.244.1.48	xen11-196

启动iptables

systemctl start iptables

确认UDP端口不通

 #nc -z -v -u 172.25.11.197 8472
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 172.25.11.197:8472.
Ncat: No route to host.

启动busybox测试，并测试

 kubectl run -i --tty busybox --image=busybox --restart=Never
 在前端执行ping

在后端执行ping

 kubectl exec busybox -- ping 10.244.0.6

测试结果是无法ping通。

1.1.1.3 通的情况

在node2上添加udp规则

iptables -I INPUT -p udp --dport 8472 -j ACCEPT

# flannel端口用于封装10.244.0.0/16网段的IP包，协议UDP，端口8472
iptables -I INPUT -p udp --dport 8472 -j ACCEPT

1.1.2开通kubectl exec端口权限

# 开通kubectl exec端口权限10250
# 如果不开通就会报如下错
 kubectl exec -it my-nginx-75897978cd-t7s6n sh
Error from server: error dialing backend: dial tcp 172.25.11.197:10250: connect: no route to host

iptables -I INPUT -p tcp --dport 10250 -j ACCEPT

# 开通后就能执行kubectl exec
# 示例修改nginx的首页内容，便于测试
# kubectl exec -it my-nginx-75897978cd-t7s6n -- bash -c "echo 10.244.2.2 >> /usr/share/nginx/html/index.html"
# 

1.2 Forward 链

具体看包封装图

由于cni0 是pod网桥IP地址，flannel.1 是路由网关地址，flannel.1 是包封装网卡地址，数据包需要Forward才能转发，打开iptables forward
注意同时在/etc/sysctl.conf 里面打开 net.ipv4.ip_forward = 1

iptables -I  FORWARD -s 10.244.0.0/16 -j ACCEPT
iptables -I  FORWARD -d 10.244.0.0/16 -j ACCEPT

这样pod之间跨主机才能互相ping通，同时宿主机才能ping通到跨主机的pod

1.3 Output 链

iptables 默认没开启OUTPUT限制，如果客户严格限制，需要加上OUTPUT过滤，需要开通10.244.0.0（pod）网段

iptables -I  OUTPUT -s 10.244.0.0/16 -j ACCEPT

2.Worker node需要配置的iptables规则

2.1 Input 规则

Protocol	iptables table	ipables chain	Port Range	Purpose	Used by
UDP	filter	INPUT	8472	flannel	network
TCP	filter	INPUT	10250	kubelet API	Control plane, Self ,kubectl exec
TCP	filter	INPUT	30000-3*****	NodePortService	All

2.2 Forward 链

具体看包封装图

由于cni0 是pod网桥IP地址，flannel.1 是路由网关地址，flannel.1 是包封装网卡地址，数据包需要Forward才能转发，打开iptables forward
注意同时在/etc/sysctl.conf 里面打开 net.ipv4.ip_forward = 1

iptables -I  FORWARD -s 10.244.0.0/16 -j ACCEPT
iptables -I  FORWARD -d 10.244.0.0/16 -j ACCEPT

这样pod之间跨主机才能互相ping通，同时宿主机才能ping通到跨主机的pod

2.3 Output 链

iptables 默认没开启OUTPUT限制，如果客户严格限制，需要加上OUTPUT过滤，需要开通10.244.0.0（pod）网段
同时需要加上连接Master node 的OUTPUT过滤

iptables -I  OUTPUT -s 10.244.0.0/16 -j ACCEPT
iptables -I  OUTPUT -d MasterIP/32 -j ACCEPT   
#如果是集群，还需要加上集群地址和其他Masternode地址

3.kubernetes NAT规则查看

kubernetes iptables NAT规则举例，这部分有kublet-proxy设置，无需自己设置

iptables -S -t nat|grep 8088
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.103.228.172/32 -p tcp -m comment --comment "default/nginx-8080: cluster IP" -m tcp --dport 8088 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.103.228.172/32 -p tcp -m comment --comment "default/nginx-8080: cluster IP" -m tcp --dport 8088 -j KUBE-SVC-LTTOZPFESA6UOUYL
iptabes -S -t nat  KUBE-MARK-MASQ

KUBE-SERVICES为 kubernets 自定义链：

[root@xen11-195 ~]# iptables -S  KUBE-SERVICES -t nat 
-N KUBE-SERVICES
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.100.43.239/32 -p tcp -m comment --comment "kube-system/traefik:http-redirect cluster IP" -m tcp --dport 1080 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.100.43.239/32 -p tcp -m comment --comment "kube-system/traefik:http-redirect cluster IP" -m tcp --dport 1080 -j KUBE-SVC-DP4KP26T3K75I2A2
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.97.140.25/32 -p tcp -m comment --comment "monitoring/alertmanager-main:web cluster IP" -m tcp --dport 9093 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.97.140.25/32 -p tcp -m comment --comment "monitoring/alertmanager-main:web cluster IP" -m tcp --dport 9093 -j KUBE-SVC-NAZP4SD6XLP35COK
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.109.140.226/32 -p tcp -m comment --comment "monitoring/prometheus-k8s:web cluster IP" -m tcp --dport 9090 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.109.140.226/32 -p tcp -m comment --comment "monitoring/prometheus-k8s:web cluster IP" -m tcp --dport 9090 -j KUBE-SVC-IFO32E4YIRUTZPGJ
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.97.96.87/32 -p tcp -m comment --comment "monitoring/prometheus-adapter:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.97.96.87/32 -p tcp -m comment --comment "monitoring/prometheus-adapter:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-GRVIJZ6QHJZF73YT
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.108.234.116/32 -p tcp -m comment --comment "default/my-nginx: cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.108.234.116/32 -p tcp -m comment --comment "default/my-nginx: cluster IP" -m tcp --dport 80 -j KUBE-SVC-BEPXDJBUHFCSYIC3
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m

iptables 实现负载均衡：

[root@xen11-195 ~]# iptables -S KUBE-SVC-LTTOZPFESA6UOUYL -t nat                
-N KUBE-SVC-LTTOZPFESA6UOUYL
-A KUBE-SVC-LTTOZPFESA6UOUYL -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-2YRI2HTTSM2JTT2G
-A KUBE-SVC-LTTOZPFESA6UOUYL -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-7TB3S7FLLR7UHYH4
-A KUBE-SVC-LTTOZPFESA6UOUYL -j KUBE-SEP-3RIUDCBRFGYQJL4F

最后追踪到pod IP地址

[root@xen11-195 ~]# iptables -S KUBE-SEP-2YRI2HTTSM2JTT2G -t nat                         
-N KUBE-SEP-2YRI2HTTSM2JTT2G
-A KUBE-SEP-2YRI2HTTSM2JTT2G -s 10.244.2.13/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-2YRI2HTTSM2JTT2G -p tcp -m tcp -j DNAT --to-destination 10.244.2.13:80
[root@xen11-195 ~]# iptables -S  KUBE-SEP-7TB3S7FLLR7UHYH4 -t nat                         
-N KUBE-SEP-7TB3S7FLLR7UHYH4
-A KUBE-SEP-7TB3S7FLLR7UHYH4 -s 10.244.2.14/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-7TB3S7FLLR7UHYH4 -p tcp -m tcp -j DNAT --to-destination 10.244.2.14:80

4. Iptables 数据流供参考