499. 【kubernetes】解决-k8s-“BGP-not-established-with”-错误

今天打开 kubernetes dashboard 仪表盘一看，发现有块红的，如下所示：

接着，通过命令行查到下面的错误：

[root@k8s0 server]#  kubectl get all -n kube-system                         
NAME                                           READY   STATUS    RESTARTS   AGE
pod/calico-kube-controllers-798cc86c47-k6x4g   1/1     Running   0          30m
pod/calico-node-cttlt                          1/1     Running   0          30m
pod/calico-node-mnp54                          1/1     Running   0          30m
pod/calico-node-smvvn                          0/1     Running   0          30m

NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/calico-node   3         3         2       3            2           kubernetes.io/os=linux   30m

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/calico-kube-controllers   1/1     1            1           30m

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/calico-kube-controllers-798cc86c47   1         1         1       30m
[root@k8s0 server]#  

进一步执行命令kubectl describe pod/calico-node-smvvn -n kube-system查到下面的错误：

calico/node is not ready: BIRD is not ready: BGP not established with 192.168.3.xxx,192.168.3.xxx

进一步执行命令 kubectl logs -f calico-node-smvvn -n kube-system（查看有问题的节点） 查到下面的日志：

2022-11-21 12:20:58.373 [INFO][98] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface nerdctl0: 10.4.0.1/24
2022-11-21 12:21:47.330 [INFO][97] felix/summary.go 100: Summarising 11 dataplane reconciliation loops over 1m3.1s: avg=5ms longest=10ms (resync-filter-v4)
2022-11-21 12:21:58.375 [INFO][98] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface nerdctl0: 10.4.0.1/24
2022-11-21 12:22:50.288 [INFO][97] felix/summary.go 100: Summarising 10 dataplane reconciliation loops over 1m3s: avg=6ms longest=13ms (resync-filter-v4)
2022-11-21 12:22:58.376 [INFO][98] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface nerdctl0: 10.4.0.1/24
2022-11-21 12:23:52.746 [INFO][97] felix/summary.go 100: Summarising 7 dataplane reconciliation loops over 1m2.5s: avg=3ms longest=3ms (resync-routes-v4,resync-routes-v4,resync-rules-v4,resync-wg)

进一步执行命令kubectl logs -f calico-node-mnp54 -n kube-system（查看没有问题的节点），日志如下

2022-11-21 12:22:58.963 [INFO][94] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface enp2s0f0: 192.168.3.102/24
2022-11-21 12:23:54.232 [INFO][96] felix/summary.go 100: Summarising 7 dataplane reconciliation loops over 1m3.9s: avg=3ms longest=3ms (resync-ipsets-v4)
2022-11-21 12:23:58.966 [INFO][94] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface enp2s0f0: 192.168.3.102/24
2022-11-21 12:24:57.809 [INFO][96] felix/summary.go 100: Summarising 8 dataplane reconciliation loops over 1m3.6s: avg=6ms longest=19ms (resync-filter-v4,resync-mangle-v4,resync-nat-v4)

• 可以看出两个pod 里显示的网段不一样，一个是10.4.0.1/24 一个是 192.168.3.102/24。

再来执行命令（ip addr）看一下有问题的那个pod对应的节点ip：

[root@k8s0 server]#  ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:92:bf:2b:20:6a brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.100/24 brd 192.168.3.255 scope global noprefixroute enp2s0f0
       valid_lft forever preferred_lft forever
    inet 192.168.3.250/24 scope global secondary enp2s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::c44d:4c26:e656:ae28/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:92:bf:2b:20:6b brd ff:ff:ff:ff:ff:ff
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 172.17.144.64/32 scope global tunl0
       valid_lft forever preferred_lft forever
6: nerdctl0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 6e:c2:85:5f:0f:20 brd ff:ff:ff:ff:ff:ff
    inet 10.4.0.1/24 brd 10.4.0.255 scope global nerdctl0
       valid_lft forever preferred_lft forever
    inet6 fe80::6cc2:85ff:fe5f:f20/64 scope link 
       valid_lft forever preferred_lft forever

• 可以看到，有问题的那个 pod 使用的是 buildkitd 创建的网卡，这是不对的。

现在知道了，这是网卡冲突导致的，我们打开 calico.yaml 文件，可以看到：

             #  Auto-detect the BGP IP address.
             - name: IP
               value: "autodetect"

• 原来BGP IP 是自动获取的。

解决办法：

修改calico.yaml 配置文件，将 IP_AUTODETECTION_METHOD 环境变量改成指定的网卡（我环境里的网卡名是：enp2s0f0），如下所示：

[root@k8s0 cni]#  kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=interface=enp2s0f0
daemonset.apps/calico-node env updated

问题解决

可以看到3个 calico-node 都恢复正常了：

---

我遇到的这个问题主要是由于：在kubernetes集群上，我还安装了用于构建镜像的 buildkiltd，它自动创建了个网卡，导致 calico 自动获取失败。也就是说最好不要再kubernetes集群上安装多余的服务（我这里的集群环境本身也不需要构建镜像，直接pull就可以了）。

**另外：**解决方法里明确指定了网卡，其实这是不好的，因为每个节点上有多个网卡时，不一定都是同一个网卡上绑定IP，如果实际情况是这样，将导致网卡再次分配失败。也就是说，最好是把多余的服务给去了，不要改calico的配置。