kubelet cgroup driver: "cgroupfs" is different from dock

最新推荐文章于 2024-08-27 11:15:11 发布

onejson

最新推荐文章于 2024-08-27 11:15:11 发布

阅读量7.2k

点赞数 1

分类专栏： k8s linux 服务器架构

本文链接：https://blog.csdn.net/wangxinxinsj/article/details/90768030

版权

架构同时被 3 个专栏收录

33 篇文章 1 订阅

订阅专栏

k8s

3 篇文章 0 订阅

订阅专栏

linux

3 篇文章 0 订阅

订阅专栏

服务器：阿里云服务器 centos7.3 64位 2cpu 2gb内存
tips: # 后面都是我输入的命令，下面是输入命令出现的提示
干跑集群的时候

 # kubeadm init --kubernetes-version="v1.14.2" --pod-network-cidr="10.244.0.0/16" --dry-run

报错，报kubelet 好像没有运行的错误，于是我就查看kubelet的状态

# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since 二 2019-06-04 11:18:42 CST; 1s ago
     Docs: https://kubernetes.io/docs/
  Process: 11375 ExecStart=/usr/bin/kubelet (code=exited, status=255)
 Main PID: 11375 (code=exited, status=255)

6月 04 11:18:42 izuf6j4eztzg3cjw0slvhhz systemd[1]: Unit kubelet.service entered failed state.
6月 04 11:18:42 izuf6j4eztzg3cjw0slvhhz systemd[1]: kubelet.service failed.

好像也看不出查错误，我再用日志命令看下

# journalctl -xeu kubelet
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: kubelet.service holdoff time over, scheduling restart.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: Starting kubelet: The Kubernetes Node Agent...
-- Subject: Unit kubelet.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has begun starting up.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.609859   11638 server.go:417] Version: v1.14.2
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.610047   11638 plugins.go:103] No cloud provider specified.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: W0604 11:19:53.610065   11638 server.go:556] standalone mode, no API client
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: W0604 11:19:53.658412   11638 server.go:474] No api server defined - no events will be sent to API server.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658435   11638 server.go:625] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658709   11638 container_manager_linux.go:261] container manager verified user specified cgroup-root exists: [
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658723   11638 container_manager_linux.go:266] Creating Container Manager object based on Node Config: {Runtim
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658840   11638 container_manager_linux.go:286] Creating device plugin manager: true
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.658879   11638 state_mem.go:36] [cpumanager] initializing new in-memory state store
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.659009   11638 state_mem.go:84] [cpumanager] updated default cpuset: ""
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.659019   11638 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.661629   11638 client.go:75] Connecting to docker on unix:///var/run/docker.sock
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.661665   11638 client.go:104] Start docker client with request timeout=2m0s
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: W0604 11:19:53.664004   11638 docker_service.go:561] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, fal
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.664038   11638 docker_service.go:238] Hairpin mode set to "hairpin-veth"
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: W0604 11:19:53.664198   11638 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.667500   11638 docker_service.go:253] Docker cri networking managed by kubernetes.io/no-op
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: I0604 11:19:53.688598   11638 docker_service.go:258] Docker Info: &{ID:HVR5:RN62:NPRF:YA4J:YBXP:E4C4:RYGP:5WVK:MGMW:TWGE:VWQE
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz kubelet[11638]: F0604 11:19:53.688710   11638 server.go:265] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgrou
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: Unit kubelet.service entered failed state.
6月 04 11:19:53 izuf6j4eztzg3cjw0slvhhz systemd[1]: kubelet.service failed.

这个命令显示的不会换行，要看后面的，用右方向键就可以了
看到这句没有

failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

告诉我们失败的原因是你的docker 运行的cgroup driver 和 kubelet 的 cgroup driver 运行的方式不是一样的，我们需要改成一样的。这里我们修改docker的。

# vim /etc/docker/daemon.json
{
      "exec-opts": ["native.cgroupdriver=cgroupfs"],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2",
      "storage-opts": [
        "overlay2.override_kernel_check=true"
      ]
    }

把native.cgroupdriver=systemd 修改成 native.cgroupdriver=cgroupfs
重启docker 并且查看docker的状态

# systemctl restart docker
# systemctl status docker
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2019-06-04 13:36:00 CST; 56s ago
     Docs: https://docs.docker.com
 Main PID: 7663 (dockerd)
   Memory: 35.7M
   CGroup: /system.slice/docker.service
           └─7663 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

docker 运行正常
我们在启动下kubelet，下面的命令不能少，配置要重载入一下，不然下面创建集群会失败

# systemctl daemon-reload
# systemctl restart kubelet
# systemctl enable kubelet
# systemctl start kubelet
# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since 二 2019-06-04 13:36:06 CST; 2min 57s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 7788 (kubelet)
   Memory: 27.9M
   CGroup: /system.slice/kubelet.service
           └─7788 /usr/bin/kubelet

再次运行干跑命令，干跑命令会告诉你创建成功，但实际创建中会报错

 # kubeadm init --kubernetes-version="v1.14.2" --pod-network-cidr="10.244.0.0/16"
[init] Using Kubernetes version: v1.14.2
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING Hostname]: hostname "izuf6j4eztzg3cjw0slvhhz" could not be reached
	[WARNING Hostname]: hostname "izuf6j4eztzg3cjw0slvhhz": lookup izuf6j4eztzg3cjw0slvhhz on 100.100.2.138:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
****
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

上面提示说，你的 kubelet 好像没有运行，kubelet 不是很健康，通过命令查看

# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: inactive (dead) since 二 2019-06-04 15:27:36 CST; 5min ago
     Docs: https://kubernetes.io/docs/
  Process: 11611 ExecStart=/usr/bin/kubelet (code=exited, status=0/SUCCESS)
 Main PID: 11611 (code=exited, status=0/SUCCESS)

6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...time.go:76
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...time.go:65
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...time.go:51
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /usr/local/go/src/runtime/panic.go:522
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...tor.go:189
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...tor.go:214
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...tor.go:125
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubern...ait.go:152
6月 04 15:27:36 izuf6j4eztzg3cjw0slvhhz systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
6月 04 15:27:36 izuf6j4eztzg3cjw0slvhhz systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Hint: Some lines were ellipsized, use -l to show in full.

什么鬼，刚才还运行正常呢，看下日志

# journalctl -xeu kubelet
6月 04 15:27:31 izuf6j4eztzg3cjw0slvhhz kubelet[11611]: /workspace/anago-v1.14.2-beta.0.85+66049e3b21efe1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8
6月 04 15:27:36 izuf6j4eztzg3cjw0slvhhz systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
-- Subject: Unit kubelet.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has begun shutting down.
6月 04 15:27:36 izuf6j4eztzg3cjw0slvhhz systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished shutting down.

没看到什么有用的信息，先根据 kubeadm init 时的错误日志来尝试下解决

[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
****
This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

上面提示到建议用 systemd 驱动,实际运行中，kubeadm init 执行后到了kubelet时
，会报超时，再去查看kubelet的状态，会发现，已经不是runngin的状态了。
利用下面的命令，把其中的systemd 也改成cgroupfs

# vim /var/lib/kubelet/kubeadm-flags.env
# vim /etc/default/kubelet

修改成一样的以后，再次执行 kubeadm init ,不会终止kubelet ，但依然会报kubelet not running 等错误
查找资料
https://github.com/kubernetes/kubeadm/issues/1315
说缺少文件

# vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

文件内容
https://github.com/kubernetes/kubernetes/blob/master/build/rpms/10-kubeadm.conf

添加内容后，重启kubelet,并再次执行kubeadm init 命令，依然报错，但kubelet 还在执行状态。

报错日志

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

最后，把服务器系统升级到了centos7.6 64位，就可以了，升级系统，升级系统，升级系统。
操作系统:centos7.6 64位
内核

# uname -a
Linux iZuf6j4eztzg3cjw0slvhhZ 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

docker

# docker info
Containers: 10
 Running: 10
 Paused: 0
 Stopped: 0
Images: 7
Server Version: 18.09.6

k8s 版本

kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

至此，k8s集群就建立成功了。其实如果版本和服务器版本能兼容的话还是挺容易的，主要是找对版本，我用的是最新的k8s版本，1.4.2的稳定版本，centos6.3 系统支持的并不是太好。