K8S 巨坑之 nodes/集群内部 无法访问ClusterIp

[root@master yaml]# kubectl get svc,pods -o wide
NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE    SELECTOR
service/db              NodePort       10.105.233.179   <none>        3306:31306/TCP                   12m    io.kompose.service=db
service/webserver-tcp   NodePort       10.98.187.153    <none>        8888:30080/TCP,10443:30443/TCP   13m    app=web-server

在worker1不但pod里面无法访问mysql(上例中的db)CLUSTER-IP，在node上ping 都不通。

[root@worker1 vagrant]# ping 10.111.182.38
PING 10.111.182.38 (10.111.182.38) 56(84) bytes of data.
^C
--- 10.111.182.38 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms

默认情况下，我们部署的kube-proxy通过查看日志，能看到如下信息：Flag proxy-mode="" unknown，assuming iptables proxy

[root@master vagrant]# kubectl get pod -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-58cc8c89f4-9b548                     1/1     Running   8          4d
coredns-58cc8c89f4-g59jz                     1/1     Running   3          4d
etcd-master.localdomain                      1/1     Running   8          4d
kube-apiserver-master.localdomain            1/1     Running   8          4d
kube-controller-manager-master.localdomain   1/1     Running   8          4d
kube-flannel-ds-ck228                        1/1     Running   3          4d
kube-flannel-ds-cxb7k                        1/1     Running   3          4d
kube-flannel-ds-jf9l6                        1/1     Running   8          4d
kube-proxy-hv5q4                             1/1     Running   0          3h15m
kube-proxy-xfww6                             1/1     Running   0          3h15m
kube-proxy-xmjc2                             1/1     Running   0          3h15m
kube-scheduler-master.localdomain            1/1     Running   8          4d

发现worker1上的错误：

[root@master vagrant]# kubectl logs kube-proxy-xmjc2 -n kube-system
W0128 06:01:38.386276       1 proxier.go:597] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0128 06:01:38.386968       1 proxier.go:597] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0128 06:01:38.387558       1 proxier.go:597] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0128 06:01:38.388179       1 proxier.go:597] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0128 06:01:38.391067       1 proxier.go:597] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0128 06:01:38.391651       1 proxier.go:597] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0128 06:01:38.392229       1 proxier.go:597] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0128 06:01:38.392789       1 proxier.go:597] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
E0128 06:01:38.393462       1 server_others.go:339] can't determine whether to use ipvs proxy, error: IPVS proxier will not be used because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh]
I0128 06:01:38.399444       1 node.go:135] Successfully retrieved node IP: 192.168.56.10
I0128 06:01:38.399469       1 server_others.go:149] **Using iptables Proxier.**
I0128 06:01:38.399654       1 server.go:529] Version: v1.16.0
I0128 06:01:38.400358       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0128 06:01:38.401607       1 config.go:313] Starting service config controller
I0128 06:01:38.401623       1 shared_informer.go:197] Waiting for caches to sync for service config
I0128 06:01:38.402022       1 config.go:131] Starting endpoints config controller
I0128 06:01:38.402046       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I0128 06:01:38.503432       1 shared_informer.go:204] Caches are synced for endpoints config 
I0128 06:01:38.503460       1 shared_informer.go:204] Caches are synced for service config 

原因分析：
并没有正确使用ipvs模式

解决方法：

1. 在master上修改kube-proxy的配置文件,添加mode 为ipvs。

[root@k8s-master ~]# kubectl edit cm kube-proxy -n kube-system

ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      strictARP: false
      syncPeriod: 30s
    kind: KubeProxyConfiguration
    metricsBindAddress: 127.0.0.1:10249
    mode: "ipvs"

在master 和worker1上，所有不正确的worker上，执行操作：

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
 #!/bin/bash 
 modprobe -- ip_vs 
 modprobe -- ip_vs_rr 
 modprobe -- ip_vs_wrr 
 modprobe -- ip_vs_sh 
 modprobe -- nf_conntrack_ipv4 
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

删除原来的POD，会自动重启kube-proxy 的pod

[root@k8s-master ~]# kubectl get pod -n kube-system | grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'

也可以单条删除

[root@master vagrant]# kubectl delete pod kube-proxy-xmjc2 -n kube-system
pod "kube-proxy-xmjc2" deleted

重新查看新生成的pod:

[root@master vagrant]# kubectl get pod -n kube-system -o wide
NAME                                         READY   STATUS    RESTARTS   AGE     IP              NODE                  NOMINATED NODE   READINESS GATES
coredns-58cc8c89f4-9b548                     1/1     Running   8          4d      10.244.0.30     master.localdomain    <none>           <none>
coredns-58cc8c89f4-g59jz                     1/1     Running   3          4d      10.244.1.21     worker2               <none>           <none>
etcd-master.localdomain                      1/1     Running   8          4d      192.168.56.9    master.localdomain    <none>           <none>
kube-apiserver-master.localdomain            1/1     Running   8          4d      192.168.56.9    master.localdomain    <none>           <none>
kube-controller-manager-master.localdomain   1/1     Running   8          4d      192.168.56.9    master.localdomain    <none>           <none>
kube-flannel-ds-ck228                        1/1     Running   3          4d      192.168.56.10   worker1.localdomain   <none>           <none>
kube-flannel-ds-cxb7k                        1/1     Running   3          4d      192.168.56.11   worker2               <none>           <none>
kube-flannel-ds-jf9l6                        1/1     Running   8          4d      192.168.56.9    master.localdomain    <none>           <none>
kube-proxy-hv5q4                             1/1     Running   0          3h17m   192.168.56.9    master.localdomain    <none>           <none>
kube-proxy-v76kd                             1/1     Running   0          3s      192.168.56.10   worker1.localdomain   <none>           <none>
kube-proxy-xfww6                             1/1     Running   0          3h17m   192.168.56.11   worker2               <none>           <none>
kube-scheduler-master.localdomain            1/1     Running   8          4d      192.168.56.9    master.localdomain    <none>           <none>

已经正常：

[root@master vagrant]# kubectl logs kube-proxy-v76kd -n kube-system
I0128 09:18:58.379053       1 node.go:135] Successfully retrieved node IP: 192.168.56.10
I0128 09:18:58.379121       1 server_others.go:176] Using ipvs Proxier.
W0128 09:18:58.379269       1 proxier.go:420] IPVS scheduler not specified, use rr by default
I0128 09:18:58.379421       1 server.go:529] Version: v1.16.0
I0128 09:18:58.380057       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0128 09:18:58.380506       1 config.go:313] Starting service config controller
I0128 09:18:58.380526       1 shared_informer.go:197] Waiting for caches to sync for service config
I0128 09:18:58.380868       1 config.go:131] Starting endpoints config controller
I0128 09:18:58.380894       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I0128 09:18:58.483264       1 shared_informer.go:204] Caches are synced for service config 
I0128 09:18:58.483655       1 shared_informer.go:204] Caches are synced for endpoints config 

从worker1中测试：

[root@worker1 vagrant]# ping 10.98.187.153
PING 10.98.187.153 (10.98.187.153) 56(84) bytes of data.
64 bytes from 10.98.187.153: icmp_seq=1 ttl=64 time=0.092 ms
64 bytes from 10.98.187.153: icmp_seq=2 ttl=64 time=0.037 ms
64 bytes from 10.98.187.153: icmp_seq=3 ttl=64 time=0.024 ms
64 bytes from 10.98.187.153: icmp_seq=4 ttl=64 time=0.089 ms
64 bytes from 10.98.187.153: icmp_seq=5 ttl=64 time=0.097 ms
64 bytes from 10.98.187.153: icmp_seq=6 ttl=64 time=0.090 ms
64 bytes from 10.98.187.153: icmp_seq=7 ttl=64 time=0.090 ms

正常了。

NND。