kubelet CNI 错误

Aug  4 14:54:54 master02 kubelet: I0804 14:54:54.755065    7867 cni.go:333] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"6fdf1de9d5ff7ec827fb1885feaed0d6a5eff049def92cc23ac5d8492be9665f\""
Aug  4 14:55:01 master02 systemd: Started Session 124 of user root.
Aug  4 14:55:23 master02 kubelet: E0804 14:55:23.756414    7867 cni.go:380] "Error deleting pod from network" err="error getting ClusterInformation: Get \"https://[10.43.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.43.0.1:443: i/o timeout" pod="default/nginx-select-master2-6cfb8f959b-kq4cv" podSandboxID={Type:docker ID:eca309ebea4ee5c77913ce74561d1f6f400e2e9069478166cb6cfb940f5136f5} podNetnsPath="" networkType="calico" networkName="k8s-pod-network"
Aug  4 14:55:23 master02 kubelet: E0804 14:55:23.757501    7867 remote_runtime.go:144] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"nginx-select-master2-6cfb8f959b-kq4cv_default\" network: error getting ClusterInformation: Get \"https://[10.43.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.43.0.1:443: i/o timeout" podSandboxID="eca309ebea4ee5c77913ce74561d1f6f400e2e9069478166cb6cfb940f5136f5"
Aug  4 14:55:23 master02 kubelet: E0804 14:55:23.757556    7867 kuberuntime_gc.go:176] "Failed to stop sandbox before removing" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"nginx-select-master2-6cfb8f959b-kq4cv_default\" network: error getting ClusterInformation: Get \"https://[10.43.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.43.0.1:443: i/o timeout" sandboxID="eca309ebea4ee5c77913ce74561d1f6f400e2e9069478166cb6cfb940f5136f5"
Aug  4 14:55:23 master02 kubelet: I0804 14:55:23.759996    7867 cni.go:333] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container \"6bcfde580efd189a096265bc0cd4eaa08832b8b324837fac69b7cf834c820f7d\""

1、重置kubernetes服务，重置网络。删除网络配置，link

1）先驱逐node节点上的pods，更具需求需不需要驱逐

~]# kubectl drain node2（node节点名称） --delete-local-data --ignore-daemonsets --force

2）重置node节点

~]# kubeadm reset
~]# systemctl stop kubelet
~]# systemctl stop docker
~]# rm -rf /var/lib/cni/
~]# rm -rf /var/lib/kubelet/*
~]# rm -rf /etc/cni/
~]# ifconfig cni0 down
~]# ifconfig flannel.1 down
~]# ifconfig docker0 down
~]# ip link delete cni0
~]# ip link delete flannel.1
根据自己的网卡名称进行操作，没有的则不需要理会

3）重新加入集群

~]# systemctl start docker 
~]# systemctl start kubelet
生成kubeadm token
~]# kubeadm  token create --print-join-command
kubeadm join master.k8s.io:16443 --token qpfi5g.sn35fgd4u1vf4l5q --discovery-token-ca-cert-hash sha256:e71a69921fd17116f687307c1ee4e5ecbdafbe724b3a40e2ed731a706ddd412a 

复制生成的token在node节点上运行，添加到集群去

4）查看node节点是否运行正常，需要等待一两分钟再查。

~]# kubectl get nodes
NAME      STATUS   ROLES                  AGE     VERSION
master1   Ready    control-plane,master   6d4h    v1.21.3
node2     Ready    <none>                 3h47m   v1.21.3
node1     Ready    <none>                 5d23h   v1.21.3

5)创建pod测试一下，本文是指定标签进行测试的，需要可以了解一下k8s 打标签

~]# cat nginx_node2.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-select-node2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        env-role: prod     #指定了标签
      containers:
      - name: nginx-select-node2
        image: nginx:latest
        ports:
        - containerPort: 80

创建

kubectl create-f nginx_node2.yaml 

查看pod

]# kubectl get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE      NOMINATED NODE   READINESS GATES
nginx-select-node2-6cfb8f959b-6ghmm   1/1     Running   0          16m   10.244.1.4    node2   <none>           <none>
nginx-select-node2-6cfb8f959b-9tsxc   1/1     Running   0          16m   10.244.1.3    node2   <none>           <none>