解决 kubernetes 重启报错的问题

参考了 如何解决 kubernetes 重启后,启来不来的问题 - jackluo - 博客园 (cnblogs.com)

由于意外突发情况，公司服务器断电关机，重启服务器后发现k8s集群起不来，有几台node节点状态是NotReady，下面展示一下吧！

1. kubectl get pods --all-namespaces

2. 显示:

3. The connection to the server 192.168.37.201:6443 was refused - did you specify the right host or port?

1. [root@k8s-node1 ~]# systemctl status kubelet

2. ● kubelet.service - kubelet: The Kubernetes Node Agent

3. Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)

4. Drop-In: /etc/systemd/system/kubelet.service.d

5. └─10-kubeadm.conf

6. Active: activating (auto-restart) (Result: exit-code) since Wed 2019-01-30 10:26:33 CST; 8s ago

7. Docs: https://kubernetes.io/docs/

8. Process: 9648 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)

9. Main PID: 9648 (code=exited, status=255)

11. Jan 30 10:26:33 k8s-node1 systemd[1]: Unit kubelet.service entered failed state.

12. Jan 30 10:26:33 k8s-node1 systemd[1]: kubelet.service failed.

这里报错显示的不明显，接下来通过这个查看：

1. 通过 journalctl -xefu kubelet 看到了

3. -- The start-up result is done.

4. Jan 30 10:27:34 k8s-node1 kubelet[9831]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

5. Jan 30 10:27:34 k8s-node1 kubelet[9831]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

6. Jan 30 10:27:34 k8s-node1 kubelet[9831]: I0130 10:27:34.877299 9831 server.go:407] Version: v1.13.2

7. Jan 30 10:27:34 k8s-node1 kubelet[9831]: I0130 10:27:34.877538 9831 plugins.go:103] No cloud provider specified.

8. Jan 30 10:27:34 k8s-node1 kubelet[9831]: I0130 10:27:34.892361 9831 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".

9. Jan 30 10:27:34 k8s-node1 kubelet[9831]: I0130 10:27:34.926248 9831 server.go:666] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /

10. Jan 30 10:27:34 k8s-node1 kubelet[9831]: F0130 10:27:34.926665 9831 server.go:261] failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename Type Size Used Priority /swapfile file 2097148 0 -2]

11. Jan 30 10:27:34 k8s-node1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a

12. Jan 30 10:27:34 k8s-node1 systemd[1]: Unit kubelet.service entered failed state.

13. Jan 30 10:27:34 k8s-node1 systemd[1]: kubelet.service failed.

上面就说得很清楚了，是没有禁用 swap 导致的；

找到原因后,解决就快了。

1. [root@k8s-node2 ~]# 关闭 Swap，机器重启后不生效

2. [root@k8s-node2 ~]# swapoff -a

3. [root@k8s-node2 ~]# cp -p /etc/fstab /etc/fstab.bak$(date '+%Y%m%d%H%M%S')

4. [root@k8s-node2 ~]# sed -i "s/\/dev\/mapper\/centos-swap/\#\/dev\/mapper\/centos-swap/g" /etc/fstab

6. [root@k8s-node2 ~]# systemctl daemon-reload

7. [root@k8s-node2 ~]# systemctl restart kubelet