检查k8s集群pod网络是否异常

事由，生产环境的k8s集群故障，orderer通信超时，kafka链接故障。对k8s高可用集群网络进行排查

Overlay 网络上意味着 NGINX Ingress Controller 需要能够将请求从NODE_1路由到NODE_2。如果 Overlay 网络不起作用，可能导致 NGINX Ingress Controller 无法将请求路由到 Pod

一、新建yml文件
vim overlaytest.yml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: overlaytest
spec:
  selector:
      matchLabels:
        name: overlaytest
  template:
    metadata:
      labels:
        name: overlaytest
    spec:
      tolerations:
      - operator: Exists
      containers:
      - image: rancherlabs/swiss-army-knife     #无网环境可能会提示无法拉取镜像。可以将导入的镜像上传至harbor，改为harbor镜像路径
        imagePullPolicy: Always
        name: overlaytest
        command: ["sh", "-c", "tail -f /dev/null"]
        terminationMessagePath: /dev/termination-log

二、启动

kubectl create -f overlaytest.yml
#查看状态
kubectl rollout status ds/overlaytest -w
返回
daemon set "overlaytest" successfully rolled out

三、编写执行脚本
确保pod正常启动后

vim 1.sh

#!/bin/bash
echo "=> Start network overlay test"
  kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.nodeName}{"\n"}{end}' |
  while read spod shost
    do kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.status.podIP}{" "}{@.spec.nodeName}{"\n"}{end}' |
    while read tip thost
      do kubectl --request-timeout='10s' exec $spod -c overlaytest -- /bin/sh -c "ping -c2 $tip > /dev/null 2>&1"
        RC=$?
        if [ $RC -ne 0 ]
          then echo FAIL: $spod on $shost cannot reach pod IP $tip on $thost
          else echo $shost can reach $thost
        fi
    done
  done
echo "=> End network overlay test"

bash 1.sh

返回正确
=> Start network overlay test
xxx cannot reach xxx
xxx cannot reach xxx
xxx cannot reach xxx
…
…
=> End network overlay test

返回错误
=> Start network overlay test
command terminated with exit code 1
xxx cannot reach xxx
…
…
=> End network overlay test

详情可以阅读官方文档
http://docs.rancher.cn/docs/rancher2.5/troubleshooting/networking/_index