1.执行kubectl get pod -o wide -n kube-system,发现有一个节点的calico-node没有出入READY状态,如下:
[root@localhost ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-78467476b-8nlm4 1/1 Running 1 42m
calico-node-7s722 0/1 Running 0 5m17s
calico-node-s4c7v 0/1 Running 0 5m17s
coredns-67c6c9cf5f-2cdfc 1/1 Running 0 3m45s
coredns-67c6c9cf5f-tr7xx 1/1 Running 0 3m45s
coredns-autoscaler-7fc7b45c69-rmpzl 1/1 Running 1 42m
metrics-server-bc9d8649-mgvnn 1/1 Running 1 42m
rke-coredns-addon-deploy-job-v7hbx 0/1 Completed 0 42m
rke-ingress-controller-deploy-job-hkm6j 0/1 Completed 0 42m
rke-metrics-addon-deploy-job-kvnl6 0/1 Completed 0 42m
rke-network-plugin-deploy-job-8rjhp 0/1 Completed 0 42m
2.执行[root@localhost ~]# kubectl describe pod/calico-node-jbcl5 -n kube-system ,发现有如下报错
calico/node is not ready: BIRD is not ready)
3.执行kubectl exec -ti calico-node-jbcl5 -n kube-system -- bash,进入calico-node,打开bird配置文件,发现router id为192.168.0.1,此IP是bride网桥地址,正常应该是ens194网卡地址:172.16.10.4,如下:
[root@localhost /]# cat /etc/calico/confd/config/bird.cfg
# Generated by confd
include "bird_aggr.cfg";
include "bird_ipam.cfg";
router id 192.168.0.1;
上面的ip是网桥IP,calico的BGP采用物理设备(网卡)作为虚拟路由器实现路由功能
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:b9:58:2b brd ff:ff:ff:ff:ff:ff
inet 172.16.10.4/24 brd 172.16.10.255 scope global noprefixroute ens192
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feb9:582b/64 scope link
valid_lft forever preferred_lft forever
4.在master执行calicoctl node status
[root@localhost ~]# wget https://github.com/projectcalico/calicoctl/releases/download/v3.5.4/calicoctl -O /usr/bin/calicoctl
[root@localhost ~]# chmod +x /usr/bin/calicoctl
[root@localhost ~]# DATASTORE_TYPE=kubernetes KUBECONFIG=~/.kube/config calicoctl get nodes
NAME
172.16.10.4
172.16.10.87
或
[root@localhost ~]# mkdir /etc/calico
[root@localhost ~]# cat /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
datastoreType: "kubernetes"
kubeconfig: "/root/.kube/config"
到此,基本可以确定是calico的BGP网卡设备识别错误导致
二、解决过程
1.查看calico官网上关于此问题的解释,如下:
官网链接:https://docs.projectcalico.org/archive/v3.18/networking/ip-autodetection
By default, Calico uses the first-found method; the first valid IP address on the first interface (excluding local interfaces such as the docker bridge). However, you can change the default method to any of the following:
Address used by the node to reach a particular IP or domain (can-reach)
Regex to include matching interfaces (interface)
Regex to exclude matching interfaces (skip-interface
大致意思是说:calico默认采用first-found方法,选择第一个接口的第一个有效IP地址(排除本地网桥接口),但是上面的情况,就是calico采用172.19.0.1网桥地址,这个比较疑惑
官网建议采用can-reach、interface、skip-inteface方式之一,通过修改 IP_AUTODETECTION_METHOD,让IP自动探测固定到某一个接口或者IP上,下面的方式采用正则表达式将接口限定在ens开头的网卡上:
kubectl set env daemonset.apps/calico-node -n calico-system IP_AUTODETECTION_METHOD=interface=ens.*
本文采用上述方式修改后并未起作用:
注:本在的k8s环境按照calico环境时采用的是calico Installation方式
后来按照官网https://docs.projectcalico.org/reference/installation/api#operator.tigera.io/v1.CalicoNetworkSpec链接说明,增加如下:
nodeAddressAutodetectionV4:
interface: ens.*
2.重新执行:kubectl create -f calico-operator.yaml kubectl create -f calico-custom-resources.yaml 问题解决
重新执行calicoctl node status