k8s node notReady之kubelet cgroup driver: “cgroupfs“ is different from docker

问题
worker节点运行正常，因为安装其他软件我就重启了一下，结果发现kubectl get nodes显示node1 notReady，这时执行kubectl describe node node1发现node1上的kubelet没有正常运行，
我手动执行systemctl start kubelet.service没有报错，但是当我执行systemctl statuskubelet.service发现服务失败了。

执行journalctl -xeu kubelet
发现具体的错误信息：
关键信息就是：

Mar 19 03:11:12 node1 kubelet[125980]: F0319 03:11:12.805339  125980 server.go:273] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

解决办法
临时将/etc/docker/daemon.json中的对应参数修改为cgroupfs

[root@node1 ~]# cat /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=cgroupfs"],
  "registry-mirrors": ["http://hub-mirror.c.163.com"]
}

[root@node1 ~]#
重启docker与kubelet
systemctl restart docker
systemctl status kubelet.service
现在kubelet正常运行了。

分析
我在另外一台worker节点执行ps -ef|grep kubelet发现两个节点启动参数竟然不同， 我当时安装时没有做什么特殊配置。
[root@node1 ~]#  systemctl status  kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Thu 2020-03-19 03:12:47 CST; 7s ago
     Docs: https://kubernetes.io/docs/
  Process: 126926 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 126926 (code=exited, status=255)

Mar 19 03:12:47 node1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Mar 19 03:12:47 node1 systemd[1]: Unit kubelet.service entered failed state.
Mar 19 03:12:47 node1 systemd[1]: kubelet.service failed.
根据启动参数$KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS 逐个分析其中不同

[root@node1 ~]# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/k                  ubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS vari                  able dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should                   use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourc                  ed from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
[root@node1 ~]# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=
[root@node1 ~]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/                  google_containers/pause:3.1"
[root@node1 ~]#

看到这里我们就直达还另外一种办法，也就是修改/var/lib/kubelet/kubeadm-flags.env文件，将其中的--cgroup-driver=cgroupfs 修改为 --cgroup-driver=systemd

办法二（也是推荐）
将/var/lib/kubelet/kubeadm-flags.env文件中的–cgroup-driver=cgroupfs 修改为 --cgroup-driver=systemd

也就是docker配置项和kubelet的一致都是systemd。