kubeadm初始化集群报错：kubelet driver: “cgroupfs“ is different from docker cgroup driver: “systemd“

kubeadm初始化集群报错：

  报错信息如下：

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
 connection refused.

	Unfortunately, an error has occurred:
		timed out waiting for the condition

	This error is likely caused by:
		- The kubelet is not running
		- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

	If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
		- 'systemctl status kubelet'
		- 'journalctl -xeu kubelet'

	Additionally, a control plane component may have crashed or exited when started by the container runtime.
	To troubleshoot, list all containers using your preferred container runtimes CLI.

	Here is one example how you may list all Kubernetes containers running in docker:
		- 'docker ps -a | grep kube | grep -v pause'
		Once you have found the failing container, you can inspect its logs with:
		- 'docker logs CONTAINERID'

  进一步查找原因：
    kubelet服务启动正常

# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Tue 2022-03-15 01:31:50 UTC; 9s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 108175 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
   Main PID: 108175 (code=exited, status=1/FAILURE)

    查询系统日志可知：“Failed to run kubelet” err="failed to run Kubelet: misconfiguration: kubelet driver: “cgroupfs” is different from docker cgroup driver: “systemd”

# journalctl -xeu kubelet
 Flag --network-plugin has been deprecated, will be removed along with dockershim.
 Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
 Flag --network-plugin has been deprecated, will be removed along with dockershim.
 Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
server.go:440] "Kubelet version" kubeletVersion="v1.22.7"
server.go:868] "Client rotation is on, will bootstrap in background"
certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
server.go:687] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
container_manager_linux.go:280] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
container_manager_linux.go:285] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
container_manager_linux.go:320] "Creating device plugin manager" devicePluginEnabled=true
state_mem.go:36] "Initialized new in-memory state store"
kubelet.go:314] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
client.go:97] "Start docker client with request timeout" timeout="2m0s"
docker_service.go:570] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
docker_service.go:263] "Docker Info" dockerInfo=&{ID:X6CY:RCM7:27L7:GQQ2:NNQZ:T2NN:CCSV:KJC2:TXO5:LRZH:EUZO:X6GA Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:7 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:false KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:25 OomKillDisable:true NGoroutines:34 SystemTime:2022-03-15T01:04:23.235473573Z LoggingDriver:json-file CgroupDriver:cgroupfs CgroupVersion:1 NEventsListener:0 KernelVersion:5.4.0-65-generic OperatingSystem:Ubuntu 20.04.2 LTS OSVersion:20.04 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc000b7e380 NCPU:2 MemTotal:16792035328 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:k8s-master Labels:[] ExperimentalBuild:false ServerVersion:20.10.13 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc Expected:2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc} RuncCommit:{ID:v1.0.3-0-gf46b6ba Expected:v1.0.3-0-gf46b6ba} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: DefaultAddressPools:[] Warnings:[WARNING: No swap limit support]}
server.go:294] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
Mar 15 01:04:23 k8s-master systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE

  解决方法：
    方法一：只修改docker配置：把docker cgroup driver: "systemd"改为kubelet driver: “cgroupfs”，请参考方法二
    方法二：修改docker和kubelet配置
      docker配置添加 “exec-opts”: [“native.cgroupdriver=systemd”]

# vim /etc/docker/daemon.json

      kubelet配置/var/lib/kubelet/config.yaml添加cgroupDriver（不同版本在哪修改，望指教）

cat > /var/lib/kubelet/config.yaml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF

      kubelet配置/var/lib/kubelet/kubeadm-flags.env，KUBELET_KUBEADM_ARGS中添加 --cgroup-driver=systemd（初始化之后就没了，不添加能成功不？望指教）

vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.5"

  重启docker和kubelet

systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet

  检查 docker info|grep “Cgroup Driver” 是否输出 Cgroup Driver: systemd

# docker info|grep "Cgroup Driver"
 Cgroup Driver: systemd

  kubeadm重置后初始化，我这里能成功