K8s问题【flannel一直重启问题,CrashLoopBackOff】

  • kubectl describe 命令查看
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  13m                   default-scheduler  Successfully assigned kube-system/kube-flannel-ds-amd64-vfhnj to node-d1
  Normal   Pulled     13m                   kubelet, node-d1   Container image "quay.io/coreos/flannel:v0.11.0-amd64" already present on machine
  Normal   Created    13m                   kubelet, node-d1   Created container install-cni
  Normal   Started    13m                   kubelet, node-d1   Started container install-cni
  Normal   Pulled     10m (x5 over 13m)     kubelet, node-d1   Container image "quay.io/coreos/flannel:v0.11.0-amd64" already present on machine
  Normal   Created    10m (x5 over 13m)     kubelet, node-d1   Created container kube-flannel
  Normal   Started    10m (x5 over 13m)     kubelet, node-d1   Started container kube-flannel
  Warning  BackOff    3m39s (x30 over 12m)  kubelet, node-d1   Back-off restarting failed container
  • kubectl logs命令查看
[root@master ~]# kubectl logs  kube-flannel-ds-amd64-9w5nq -n kube-system
I1216 05:59:40.055608       1 main.go:527] Using interface with name eth0 and address 192.168.1.82
I1216 05:59:40.055666       1 main.go:544] Defaulting external address to interface address (192.168.1.82)
E1216 06:00:10.056546       1 main.go:241] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-amd64-9w5nq': Get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-amd64-9w5nq: dial tcp 10.96.0.1:443: i/o timeout

问题排查

  • 网络问题?

通过curl命令测试,网络没有问题。

  • 测试kube-proxy

重启该节点上的kube-proxy容器,并查看日志

kubectl delete pod kube-proxy-vtd27 -n kube-system
[root@master ~]# kubectl logs kube-proxy-pljct -n kube-system
W1216 06:30:51.741835       1 proxier.go:513] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1216 06:30:51.742536       1 proxier.go:513] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1216 06:30:51.748495       1 proxier.go:513] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1216 06:30:51.749223       1 proxier.go:513] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
E1216 06:30:51.750805       1 server_others.go:259] can't determine whether to use ipvs proxy, error: IPVS proxier will not be used because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh]
I1216 06:30:51.757054       1 server_others.go:143] Using iptables Proxier.
W1216 06:30:51.757122       1 proxier.go:321] clusterCIDR not specified, unable to distinguish between internal and external traffic
I1216 06:30:51.757338       1 server.go:534] Version: v1.15.0
  • 这里明显有个问题。一些需要的内核模块加载失败,参考安装文档,已经配置过内核模块。重新尝试,问题依然存在。
[root@master ~]# cat > /etc/sysconfig/modules/ipvs.modules <<EOF
> #!/bin/bash
> modprobe -- ip_vs
> modprobe -- ip_vs_rr
> modprobe -- ip_vs_wrr
> modprobe -- ip_vs_sh
> modprobe -- nf_conntrack_ipv4
> EOF
[root@master ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
ip_vs_sh               12688  0 
ip_vs_wrr              12697  0 
ip_vs_rr               12600  150 
ip_vs                 145497  156 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack_ipv4      15053  6 
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
nf_conntrack          139224  9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c              12644  3 ip_vs,nf_nat,nf_conntrack
[root@master ~]# lsmod | grep -e ip_vs -e nf_conntrack_ipv4
ip_vs_sh               12688  0 
ip_vs_wrr              12697  0 
ip_vs_rr               12600  150 
ip_vs                 145497  156 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack_ipv4      15053  6 
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
nf_conntrack          139224  9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c              12644  3 ip_vs,nf_nat,nf_conntrack
  • 问题解决

由于 ipvs 已经加入到内核主干,所以需要内核模块支持,请确保内核已经加载了相应模块;如不确定,执行以下脚本,以确保内核加载相应模块,否则会出现 failed to load kernel modules: [ip_vs_rr ip_vs_sh ip_vs_wrr] 错误

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack_ipv4"
for kernel_module in \${ipvs_modules}; do
    /sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        /sbin/modprobe \${kernel_module}
    fi
done
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs

执行后应该如下图所示,如果 lsmod | grep ip_vs 并未出现 ip_vs_rr 等模块;

[root@node-d1 ~]# lsmod | grep ip_vs
ip_vs_ftp              13079  0 
ip_vs_sed              12519  0 
ip_vs_nq               12516  0 
ip_vs_sh               12688  0 
ip_vs_dh               12688  0 
ip_vs_lblcr            12922  0 
ip_vs_lblc             12819  0 
ip_vs_wrr              12697  0 
ip_vs_wlc              12519  0 
ip_vs_lc               12516  0 
nf_nat                 26583  5 ip_vs_ftp,nf_nat_ipv4,nf_nat_ipv6,xt_nat,nf_nat_masquerade_ipv4
ip_vs_rr               12600  136 
ip_vs                 145497  162 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_lblcr,ip_vs_lblc
nf_conntrack          139224  9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c              12644  3 ip_vs,nf_nat,nf_conntrack

 最后重启Kube-proxyflannel容器,问题解决。

  • 2
    点赞
  • 14
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
您好!kube-flannel状态CrashLoopBackOff通常表示kube-flannel容器正在启动时遇到问题并崩溃。引起此问题的原因可能有多种,下面是一些可能的解决方法: 1. 检查kube-flannel日志:您可以通过kubectl logs命令检查kube-flannel容器的日志,以了解更多关于崩溃的详细信息。例如: ``` kubectl logs -n kube-system <kube-flannel-pod-name> ``` 2. 检查网络配置:确保您的Kubernetes集群的网络配置正确无误。kube-flannel与网络相关,因此如果网络配置有问题,可能会导致容器崩溃。确保您的网络插件正确安装和配置,并且与其他组件兼容。 3. 检查资源限制:kube-flannel容器可能由于资源限制不足而崩溃。请确保为kube-flannel分配了足够的资源(CPU和内存),以满足其运行需求。 4. 检查kube-flannel版本和Kubernetes版本的兼容性:确保您使用的kube-flannel版本与您的Kubernetes版本兼容。某些kube-flannel版本可能不支持特定的Kubernetes版本,这可能导致容器崩溃。 5. 重新安装kube-flannel:如果上述方法都没有解决问题,可以尝试重新安装kube-flannel。首先,删除现有的kube-flannel部署,然后再次安装它。可以使用以下命令来完成这些操作: ``` kubectl delete daemonset kube-flannel -n kube-system kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ``` 希望这些解决方法对您有所帮助!如果您有其他问题,请随时提问。
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值