报错说明
-
算是记录一个坑吧,花了我挺多时间解决这个破问题的。
-
今天给集群的node节点配置一个–insecure-registry的IP,配置内容如下
[root@node1 ~]# cat /usr/lib/systemd/system/docker.service | grep ExecStart
#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecStart=/usr/bin/dockerd --insecure-registry=192.168.59.151:5000 -H fd:// --containerd=/run/containerd/containerd.sock
[root@node1 ~]#
- 然后启动就报错了,docker状态和报错内容如下
#重启报错
[root@node1 ~]# systemctl restart docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
[root@node1 ~]#
# 状态下的日志
[root@node1 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Tue 2021-11-09 15:47:44 CST; 13s ago
Docs: https://docs.docker.com
Process: 38530 ExecStart=/usr/bin/dockerd --insecure-registry=192.168.59.142:5000 -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 38530 (code=exited, status=1/FAILURE)
Nov 09 15:47:42 node1 systemd[1]: docker.service failed.
Nov 09 15:47:44 node1 systemd[1]: docker.service holdoff time over, scheduling restart.
Nov 09 15:47:44 node1 systemd[1]: Stopped Docker Application Container Engine.
Nov 09 15:47:44 node1 systemd[1]: start request repeated too quickly for docker.service
Nov 09 15:47:44 node1 systemd[1]: Failed to start Docker Application Container Engine.
Nov 09 15:47:44 node1 systemd[1]: Unit docker.service entered failed state.
Nov 09 15:47:44 node1 systemd[1]: docker.service failed.
Nov 09 15:47:44 node1 systemd[1]: start request repeated too quickly for docker.service
Nov 09 15:47:44 node1 systemd[1]: Failed to start Docker Application Container Engine.
Nov 09 15:47:44 node1 systemd[1]: docker.service failed.
[root@node1 ~]#
#我们再来看messages
[root@node1 ~]# tail -n100 /var/log/messages
Nov 9 15:49:01 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:49:01 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:49:01 node1 systemd: Started Kubernetes systemd probe.
Nov 9 15:49:01 node1 kubelet: I1109 15:49:01.272480 38654 server.go:440] "Kubelet version" kubeletVersion="v1.21.0"
Nov 9 15:49:01 node1 kubelet: I1109 15:49:01.272947 38654 server.go:851] "Client rotation is on, will bootstrap in background"
Nov 9 15:49:01 node1 kubelet: I1109 15:49:01.291828 38654 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 9 15:49:01 node1 kubelet: I1109 15:49:01.295753 38654 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.338147 38654 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.338751 38654 container_manager_linux.go:278] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.338938 38654 container_manager_linux.go:283] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.338984 38654 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.339007 38654 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.339018 38654 container_manager_linux.go:319] "Creating device plugin manager" devicePluginEnabled=true
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.339254 38654 kubelet.go:310] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.339308 38654 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Nov 9 15:49:06 node1 kubelet: I1109 15:49:06.339330 38654 client.go:97] "Start docker client with request timeout" timeout="2m0s"
Nov 9 15:49:06 node1 kubelet: E1109 15:49:06.339673 38654 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Nov 9 15:49:06 node1 systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 9 15:49:06 node1 systemd: Unit kubelet.service entered failed state.
Nov 9 15:49:06 node1 systemd: kubelet.service failed.
Nov 9 15:49:16 node1 systemd: kubelet.service holdoff time over, scheduling restart.
Nov 9 15:49:16 node1 systemd: Stopped kubelet: The Kubernetes Node Agent.
Nov 9 15:49:16 node1 systemd: Started kubelet: The Kubernetes Node Agent.
Nov 9 15:49:16 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:49:16 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:49:16 node1 systemd: Started Kubernetes systemd probe.
Nov 9 15:49:16 node1 kubelet: I1109 15:49:16.678405 38676 server.go:440] "Kubelet version" kubeletVersion="v1.21.0"
Nov 9 15:49:16 node1 kubelet: I1109 15:49:16.678871 38676 server.go:851] "Client rotation is on, will bootstrap in background"
Nov 9 15:49:16 node1 kubelet: I1109 15:49:16.698382 38676 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 9 15:49:16 node1 kubelet: I1109 15:49:16.700415 38676 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.738824 38676 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.739317 38676 container_manager_linux.go:278] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.739470 38676 container_manager_linux.go:283] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.739516 38676 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.739535 38676 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.739545 38676 container_manager_linux.go:319] "Creating device plugin manager" devicePluginEnabled=true
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.739782 38676 kubelet.go:310] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.739834 38676 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Nov 9 15:49:21 node1 kubelet: I1109 15:49:21.739860 38676 client.go:97] "Start docker client with request timeout" timeout="2m0s"
Nov 9 15:49:21 node1 kubelet: E1109 15:49:21.740326 38676 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Nov 9 15:49:21 node1 systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 9 15:49:21 node1 systemd: Unit kubelet.service entered failed state.
Nov 9 15:49:21 node1 systemd: kubelet.service failed.
Nov 9 15:49:31 node1 systemd: kubelet.service holdoff time over, scheduling restart.
Nov 9 15:49:31 node1 systemd: Stopped kubelet: The Kubernetes Node Agent.
Nov 9 15:49:31 node1 systemd: Started kubelet: The Kubernetes Node Agent.
Nov 9 15:49:32 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:49:32 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:49:32 node1 systemd: Started Kubernetes systemd probe.
Nov 9 15:49:32 node1 kubelet: I1109 15:49:32.378876 38706 server.go:440] "Kubelet version" kubeletVersion="v1.21.0"
Nov 9 15:49:32 node1 kubelet: I1109 15:49:32.379407 38706 server.go:851] "Client rotation is on, will bootstrap in background"
Nov 9 15:49:32 node1 kubelet: I1109 15:49:32.398293 38706 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 9 15:49:32 node1 kubelet: I1109 15:49:32.466688 38706 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.502634 38706 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.503144 38706 container_manager_linux.go:278] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.503303 38706 container_manager_linux.go:283] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.503348 38706 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.503367 38706 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.503377 38706 container_manager_linux.go:319] "Creating device plugin manager" devicePluginEnabled=true
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.503602 38706 kubelet.go:310] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.503654 38706 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Nov 9 15:49:37 node1 systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 9 15:49:37 node1 kubelet: I1109 15:49:37.503679 38706 client.go:97] "Start docker client with request timeout" timeout="2m0s"
Nov 9 15:49:37 node1 kubelet: E1109 15:49:37.504111 38706 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Nov 9 15:49:37 node1 systemd: Unit kubelet.service entered failed state.
Nov 9 15:49:37 node1 systemd: kubelet.service failed.
Nov 9 15:49:47 node1 systemd: kubelet.service holdoff time over, scheduling restart.
Nov 9 15:49:47 node1 systemd: Stopped kubelet: The Kubernetes Node Agent.
Nov 9 15:49:47 node1 systemd: Started kubelet: The Kubernetes Node Agent.
Nov 9 15:49:47 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:49:47 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:49:47 node1 systemd: Started Kubernetes systemd probe.
Nov 9 15:49:47 node1 kubelet: I1109 15:49:47.820327 38730 server.go:440] "Kubelet version" kubeletVersion="v1.21.0"
Nov 9 15:49:47 node1 kubelet: I1109 15:49:47.820782 38730 server.go:851] "Client rotation is on, will bootstrap in background"
Nov 9 15:49:47 node1 kubelet: I1109 15:49:47.839802 38730 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 9 15:49:47 node1 kubelet: I1109 15:49:47.912499 38730 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.950362 38730 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.950843 38730 container_manager_linux.go:278] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.950989 38730 container_manager_linux.go:283] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.951032 38730 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.951049 38730 container_manager_linux.go:314] "Initializing Topology Manager" policy="none" scope="container"
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.951060 38730 container_manager_linux.go:319] "Creating device plugin manager" devicePluginEnabled=true
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.951271 38730 kubelet.go:310] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.951320 38730 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Nov 9 15:49:52 node1 systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 9 15:49:52 node1 kubelet: I1109 15:49:52.951342 38730 client.go:97] "Start docker client with request timeout" timeout="2m0s"
Nov 9 15:49:52 node1 kubelet: E1109 15:49:52.951825 38730 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Nov 9 15:49:52 node1 systemd: Unit kubelet.service entered failed state.
Nov 9 15:49:52 node1 systemd: kubelet.service failed.
Nov 9 15:50:01 node1 systemd: Started Session 22 of user root.
Nov 9 15:50:03 node1 systemd: kubelet.service holdoff time over, scheduling restart.
Nov 9 15:50:03 node1 systemd: Stopped kubelet: The Kubernetes Node Agent.
Nov 9 15:50:03 node1 systemd: Started kubelet: The Kubernetes Node Agent.
Nov 9 15:50:03 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:50:03 node1 kubelet: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Nov 9 15:50:03 node1 systemd: Started Kubernetes systemd probe.
Nov 9 15:50:03 node1 kubelet: I1109 15:50:03.310912 38765 server.go:440] "Kubelet version" kubeletVersion="v1.21.0"
Nov 9 15:50:03 node1 kubelet: I1109 15:50:03.311529 38765 server.go:851] "Client rotation is on, will bootstrap in background"
Nov 9 15:50:03 node1 kubelet: I1109 15:50:03.330817 38765 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 9 15:50:03 node1 kubelet: I1109 15:50:03.332741 38765 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
[root@node1 ~]#
原因及处理方法
- 原因就是:
如果不出意外,你应该配置了/etc/docker/daemon.json
这个文件的,就是因为这个里面的内容和上面配置的那个值冲突了【这个配置私有仓库的,感兴趣的小伙伴可以在我文章中搜一下“私有”,会出来搭建私有仓库的教程,里面有说明哈】
[root@node1 ~]# cat /etc/docker/daemon.json
{
"insecure-registries":["192.168.159.129"]
}
[root@node1 ~]#
- 处理方式
我们可以把/etc/docker/daemon.json
这个文件中配置成阿里云的加速器就好了呀~,也可以直接删除。
还是配置成下面的阿里云加速器吧,没坏处。。。
[root@node1 ~]# cat /etc/docker/daemon.json
{
"registry-mirrors":["https://frz7i079.mirror.aliyuncs.com"]
}
[root@node1 ~]#
- 然后重启服务,就正常了呀~
[root@node1 ~]# systemctl daemon-reload ; systemctl restart docker ; systemctl is-active docker
active
[root@node1 ~]#