The connection to the server 127.0.0.1:11081 was refused - did you specify the right host or port?

问题

k8s集群重启之后出现如下报错

The connection to the server 127.0.0.1:11081 was refused - did you specify the right host or port?

分析问题

查看kubelet的状态

# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-dce.conf
        /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: failed (Result: start-limit) since Fri 2021-07-16 09:59:14 HKT; 20min ago
     Docs: https://kubernetes.io/docs/
  Process: 2663 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 2663 (code=exited, status=255)

Jul 16 09:59:04 dce-192-168-153-140 systemd[1]: Unit kubelet.service entered failed state.
Jul 16 09:59:04 dce-192-168-153-140 systemd[1]: kubelet.service failed.
Jul 16 09:59:14 dce-192-168-153-140 systemd[1]: kubelet.service holdoff time over, schedul...t.
Jul 16 09:59:14 dce-192-168-153-140 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jul 16 09:59:14 dce-192-168-153-140 systemd[1]: start request repeated too quickly for kub...ce
Jul 16 09:59:14 dce-192-168-153-140 systemd[1]: Failed to start kubelet: The Kubernetes No...t.
Jul 16 09:59:14 dce-192-168-153-140 systemd[1]: Unit kubelet.service entered failed state.
Jul 16 09:59:14 dce-192-168-153-140 systemd[1]: kubelet.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

    /usr/lib/systemd/system/kubelet.service.d
       └─10-kubeadm.conf

kubelet drop-in了kubeadm的配置文件

查看日志

-- The start-up result is done.
Jul 17 10:07:26 dce-192-168-153-140 kubelet[2615]: F0717 10:07:26.426856    2615 server.go:199] failed to load Kubelet config file /var/lib/kubelet/conf
Jul 17 10:07:26 dce-192-168-153-140 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Jul 17 10:07:26 dce-192-168-153-140 systemd[1]: Unit kubelet.service entered failed state.
Jul 17 10:07:26 dce-192-168-153-140 systemd[1]: kubelet.service failed.
Jul 17 10:07:36 dce-192-168-153-140 systemd[1]: kubelet.service holdoff time over, scheduling restart.
Jul 17 10:07:36 dce-192-168-153-140 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
Jul 17 10:07:36 dce-192-168-153-140 systemd[1]: start request repeated too quickly for kubelet.service
Jul 17 10:07:36 dce-192-168-153-140 systemd[1]: Failed to start kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has failed
-- Defined-By: systemd

查看日志得知是因为安装了kubeadm导致冲突
failed to load Kubelet config file /var/lib/kubelet/conf

解决方法

删除/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf文件

[root@dce-192-168-153-140 system]# cd /usr/lib/systemd/system/kubelet.service.d/
[root@dce-192-168-153-140 kubelet.service.d]# ls
10-kubeadm.conf
[root@dce-192-168-153-140 kubelet.service.d]# rm -rf 10-kubeadm.conf 
[root@dce-192-168-153-140 kubelet.service.d]# systemctl daemon-reload 
[root@dce-192-168-153-140 kubelet.service.d]# systemctl restart kubelet
[root@dce-192-168-153-140 kubelet.service.d]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-dce.conf
   Active: active (running) since Sat 2021-07-17 11:23:19 HKT; 6s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 3804 (kubelet)
    Tasks: 17
   Memory: 20.0M
   CGroup: /system.slice/kubelet.service
           └─3804 /usr/bin/kubelet -v=2 --node-ip 192.168.153.140 --pod-infra-container-image 192.168.153.139/kube-system/dce-kube-pause:3.1 --kubeco...

Jul 17 11:23:20 dce-192-168-153-140 kubelet[3804]: E0717 11:23:20.236149    3804 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:5... refused
Jul 17 11:23:21 dce-192-168-153-140 kubelet[3804]: E0717 11:23:21.144116    3804 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:5... refused
Jul 17 11:23:21 dce-192-168-153-140 kubelet[3804]: E0717 11:23:21.663057    3804 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:5... refused
Jul 17 11:23:21 dce-192-168-153-140 kubelet[3804]: E0717 11:23:21.674050    3804 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/config/apise... refused
Jul 17 11:23:22 dce-192-168-153-140 kubelet[3804]: I0717 11:23:22.178060    3804 certificate_manager.go:409] Rotating certificates
Jul 17 11:23:22 dce-192-168-153-140 kubelet[3804]: E0717 11:23:22.179723    3804 certificate_manager.go:434] Failed while requesting a signed certifi...
Jul 17 11:23:23 dce-192-168-153-140 kubelet[3804]: E0717 11:23:23.091095    3804 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:5... refused
Jul 17 11:23:24 dce-192-168-153-140 kubelet[3804]: E0717 11:23:24.410211    3804 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/config/apise... refused
Jul 17 11:23:24 dce-192-168-153-140 kubelet[3804]: E0717 11:23:24.714123    3804 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:5... refused
Jul 17 11:23:25 dce-192-168-153-140 kubelet[3804]: W0717 11:23:25.216282    3804 cni.go:237] Unable to update cni config: no networks found i...ni/net.d
Hint: Some lines were ellipsized, use -l to show in full.