故障分析：部署在某些Node上的所有Pod无法访问svc

相关版本

类型	版本号
kubelet	v1.22.0
操作系统	CentOS Linux release 7.9.2009 (Core)
内核版本	3.10.0-1160.108.1.el7.x86_64

       

故障现象

        部署在某几个Node上的所有Pod无法访问svc:端口，使用telnet访问超时，使用nc -vz也不通，但在其他Node上的Pod可以正常访问

---

排查思路

步骤一

        因为在其他Node上的Pod可以访问到svc，则确认svc以及对应Pod无任何问题。尝试将无法访问到svc的Node上的Pod调度到其他Node节点上

        方法1 将node设置为不可调用，然后删除掉需要调度至其他节点的pod

kubectl cordon node-5

kubectl -n web delete po web-32dqq23w --force

kubectl uncordon node-5

        方法2 直接修改nodeName字段

    spec:
      nodeName: node-5
      serviceAccountName: telnet-access
      containers:
      - name: telnet-container
        image: registry.cn-hongkong.aliyuncs.com/da/centos-tool:v1

 步骤二

        将Pod调度至其他节点之后，发现调度过去的Pod可以正常访问到svc，定位到是node节点的问,查看Node问题节点的kube-proxy、calico-node、kubelet，以及core-dns日志

kubectl -n kube-system logs kube-proxy-xmjbq    #查看kube-proxy日志

kubectl -n calico-system logs calico-node-b4d2t    #查看calico-node日志

journalctl -u kubelet     #查看对应node节点的kubelet日志

kubectl -n kube-system logs coredns-78fcd69978-sk6r9    #查看coredns日志

kubelet日志报错

-- Logs begin at Sun 2024-03-10 00:12:24 CST, end at Mon 2024-03-18 11:54:47 CST. --
Mar 10 00:12:57 master systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Mar 10 00:12:57 master kubelet-pre-start.sh[786]: modprobe: ERROR: could not insert 'bridge': Unknown symbol in module, or unknown parameter (see dmesg)
Mar 10 00:12:57 master kubelet-pre-start.sh[786]: modprobe: ERROR: could not insert 'ip_vs_rr': Unknown symbol in module, or unknown parameter (see dmesg)
Mar 10 00:12:57 master kubelet-pre-start.sh[786]: * Applying /usr/lib/sysctl.d/00-system.conf ...
Mar 10 00:12:57 master kubelet-pre-start.sh[786]: net.bridge.bridge-nf-call-ip6tables = 0
Mar 10 00:12:57 master kubelet-pre-start.sh[786]: net.bridge.bridge-nf-call-iptables = 0
Mar 10 00:12:57 master kubelet-pre-start.sh[786]: net.bridge.bridge-nf-call-arptables = 0

日志显示在启动kubelet服务的过程中遇到了一些问题,主要有:

1. 加载bridge和ip_vs_rr内核模块失败,提示"Unknown symbol in module, or unknown parameter"。这通常意味着内核版本与模块不匹配,或者系统缺少必要的依赖。

2. 应用sysctl配置/usr/lib/sysctl.d/00-system.conf时,将以下参数设置为了0:

   • net.bridge.bridge-nf-call-ip6tables
   • net.bridge.bridge-nf-call-iptables
   • net.bridge.bridge-nf-call-arptables

kube-proxy日志报错

[root@master ]# kubectl -n kube-system logs kube-proxy-t9lfg 
I0318 04:32:12.118231       1 node.go:172] Successfully retrieved node IP: 10.0.2.188
I0318 04:32:12.118452       1 server_others.go:140] Detected node IP 10.0.2.188
I0318 04:32:12.229846       1 server_others.go:206] kube-proxy running in dual-stack mode, IPv4-primary
I0318 04:32:12.229946       1 server_others.go:212] Using iptables Proxier.
I0318 04:32:12.229985       1 server_others.go:219] creating dualStackProxier for iptables.
W0318 04:32:12.230051       1 server_others.go:495] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
I0318 04:32:12.230591       1 proxier.go:275] "Missing br-netfilter module or unset sysctl br-nf-call-iptables; proxy may not work as intended"
I0318 04:32:12.230900       1 proxier.go:275] "Missing br-netfilter module or unset sysctl br-nf-call-iptables; proxy may not work as intended"
I0318 04:32:12.231553       1 server.go:649] Version: v1.22.0
I0318 04:32:12.253605       1 conntrack.go:52] Setting nf_conntrack_max to 262144
I0318 04:32:12.254362       1 config.go:224] Starting endpoint slice config controller
I0318 04:32:12.254427       1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I0318 04:32:12.254527       1 config.go:315] Starting service config controller
I0318 04:32:12.254549       1 shared_informer.go:240] Waiting for caches to sync for service config
E0318 04:32:12.263180       1 event_broadcaster.go:253] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"node-5.17bdc1a15c5ea1f7", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, EventTime:v1.MicroTime{Time:time.Time{wall:0xc1760ed30f277a9c, ext:258460093, loc:(*time.Location)(0x2d81340)}}, Series:(*v1.EventSeries)(nil), ReportingController:"kube-proxy", ReportingInstance:"kube-proxy-node-5", Action:"StartKubeProxy", Reason:"Starting", Regarding:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"node-5", UID:"node-5", APIVersion:"", ResourceVersion:"", FieldPath:""}, Related:(*v1.ObjectReference)(nil), Note:"", Type:"Normal", DeprecatedSource:v1.EventSource{Component:"", Host:""}, DeprecatedFirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeprecatedLastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeprecatedCount:0}': 'Event "node-5.17bdc1a15c5ea1f7" is invalid: involvedObject.namespace: Invalid value: "": does not match event.namespace' (will not retry!)
I0318 04:32:12.359129       1 shared_informer.go:247] Caches are synced for endpoint slice config 
I0318 04:32:12.359422       1 shared_informer.go:247] Caches are synced for service config 

        日志显示缺少 br-netfilter 模块或未设置 br-nf-call-iptables 的 sysctl,代理可能无法按预期工作

---

解决方案

1. 确保系统内核版本与kubelet及相关组件兼容

2. 检查系统是否安装了必要的内核模块和依赖包,如br_netfilter等。

lsmod | grep br_netfilter
lsmod | grep ip_vs_rr

3. 在启动kubelet之前,手动加载需要的内核模块:

modprobe br_netfilter
modprobe ip_vs_rr

4. 修改/etc/sysctl.conf或/etc/sysctl.d/k8s.conf,添加以下配置:

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1

    使其生效

sysctl -p

   查看参数

sysctl net.bridge.bridge-nf-call-ip6tables
sysctl net.bridge.bridge-nf-call-iptables
sysctl net.bridge.bridge-nf-call-arptables

        这些参数控制着bridge数据包是否经过iptables/ip6tables/arptables处理。某些CNI网络插件如flannel、calico等需要将其设置为1，这里往往可能是calico或flannel等CNI网络插件初始化的时候配置失败导致

5. 重启或重建对应Node的kubelet、calico-node pod、kube-proxy pod

systemctl restart kubelet

kubectl -n kube-system delete po kube-proxy-t9lfg

kubectl -n calico-system delete po calico-node-xgz7l

附上一个可用于检测这种情况的daemonset部署的yaml

执行之后会在每个node节点上部署一个pod，每10分钟在每个node节点上通过telnet命令检测 test命名空间下的所有svc+端口能否连通，如若有需要跳过检测的svc可以在configmap telnet-exclude-services中配置

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: telnet-access
  namespace: test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: test
  name: telnet-role
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: telnet-rolebinding
  namespace: test
subjects:
- kind: ServiceAccount
  name: telnet-access
  namespace: test
roleRef:
  kind: Role
  name: telnet-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: telnet-script
  namespace: test
data:
  telnet-script.sh: |
    #!/bin/bash
    while true; do
      NODE_NAME=$(echo $MY_NODE_NAME)
      CURRENT_TIME=$(date +"%Y-%m-%d %H:%M")
      # 使用 Kubernetes API 获取服务列表
      SERVICES=$(curl -sSk -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/test/services)
      # 解析服务名称和端口
      echo $SERVICES | jq -r '.items[] | .metadata.name + " " + (.spec.ports[] | .port | tostring)' | while read svc port; do
        if ! grep -q "$svc:$port" /exclude-services.txt; then
          timeout 3 bash -c "cat < /dev/null > /dev/tcp/$svc/$port" 2> /dev/null
          if [ $? -ne 0 ]; then
            echo "$CURRENT_TIME $NODE_NAME $svc port $port is not open"
          fi
        fi
      done
      sleep 600  # 每10分钟运行一次
    done
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: telnet-exclude-services
  namespace: test
data:
  exclude-services.txt: |
    svc-nacos:19200
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: telnet-daemonset
  namespace: test
spec:
  selector:
    matchLabels:
      app: telnet
  template:
    metadata:
      labels:
        app: telnet
    spec:
      serviceAccountName: telnet-access
      containers:
      - name: telnet-container
        image: registry.cn-hongkong.aliyuncs.com/dawncloudpublic/centos-tool:v1  #此处的镜像需要有jq、telnet、curl命令
        volumeMounts:
        - name: telnet-script-volume
          mountPath: /scripts
        - name: exclude-services-volume
          mountPath: /exclude-services.txt
          subPath: exclude-services.txt
        - name: host-timezone
          mountPath: /etc/localtime
          readOnly: true
        command: ["/bin/bash"]
        args: ["/scripts/telnet-script.sh"]
        env:
        - name: MY_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
      volumes:
      - name: telnet-script-volume
        configMap:
          name: telnet-script
      - name: exclude-services-volume
        configMap:
          name: telnet-exclude-services
      - name: host-timezone
        hostPath:
          path: /etc/localtime
          type: File

部署之后执行以下命令即可

kubectl -n test get po|grep telnet|awk '{print $1}'|xargs -I {} kubectl -n test logs {}