问题出现场景
在lxd容器中创建了docker容器,有一天所有lxd容器内的docker容器全都起不来;启动时报错
root@bb:~# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2021-01-27 20:20:21 CST; 52s ago
Docs: https://docs.docker.com
Process: 435 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 435 (code=exited, status=1/FAILURE)
Jan 27 20:20:21 bb systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Jan 27 20:20:21 bb systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jan 27 20:20:21 bb systemd[1]: Stopped Docker Application Container Engine.
Jan 27 20:20:21 bb systemd[1]: docker.service: Start request repeated too quickly.
Jan 27 20:20:21 bb systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 27 20:20:21 bb systemd[1]: Failed to start Docker Application Container En
查看日志详情:
root@bb:~# sudo journalctl -xe
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has begun starting up.
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.784691543+08:00" level=info msg="systemd-resolved is running, so using resolvconf: /run/systemd/resolve/resolv.conf"
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787316279+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787383289+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787547821+08:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787583664+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787577781+08:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787636606+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787714113+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42007f440, CONNECTING" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787750049+08:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787786012+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.787876910+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4207f54e0, CONNECTING" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.788110267+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42007f440, READY" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.788142830+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4207f54e0, READY" module=grpc
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.792378731+08:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support l
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.795697602+08:00" level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.798731718+08:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support l
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.835798600+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.836420988+08:00" level=warning msg="Your kernel does not support swap memory limit"
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.836574929+08:00" level=warning msg="Your kernel does not support cgroup rt period"
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.836612848+08:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.836650679+08:00" level=warning msg="Your kernel does not support cgroup blkio weight"
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.836687231+08:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.838965475+08:00" level=info msg="Loading containers: start."
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.842786432+08:00" level=warning msg="Running modprobe nf_nat failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.846767605+08:00" level=warning msg="Running modprobe xt_conntrack failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() cou
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.931682527+08:00" level=warning msg="Enabling IP forwarding failed: open /proc/sys/net/ipv4/ip_forward: read-only file system"
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.933652825+08:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Jan 27 20:21:40 bb dockerd[626]: Error starting daemon: Error initializing network controller: error obtaining controller instance: Enabling IP forwarding failed: open /proc/sys/net/ipv4/ip_forward: read-only file
Jan 27 20:21:40 bb dockerd[626]: time="2021-01-27T20:21:40.934417502+08:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Jan 27 20:21:40 bb systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jan 27 20:21:40 bb systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 27 20:21:40 bb systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
上面日志中有一个关键错误语句:
controller: error obtaining controller instance: Enabling IP forwarding failed: open /proc/sys/net/ipv4/ip_forward: read-only file
得出问题应该是该文件不能修改导致的,查询宿主机文件内容,发现该文件内容为0,要改成1(原因参考:https://blog.csdn.net/li_101357/article/details/78415461)
解决办法
在宿主机中修改/proc/sys/net/ipv4/ip_forward内容为1
echo 1 > /proc/sys/net/ipv4/ip_forward
注意:将此属性设置为0时,使用sudo设置不成功时,切换到root可以修改;
修改完成后重启lxd容器,发现docker起来了。