getting the final child‘s pid from pipe caused \“EOF\““: unknown

k8s pod一直处于ContainerCreating，kd查看事件中报错

 Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "image-nginx": Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\"": unknown

 

查看该pod所在的服务器docker状态，里面包含日志

Jan 08 07:23:07 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:07.977541803Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 08 07:23:07 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:07.983042775Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 08 07:23:08 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:08.116070667Z" level=error msg="b9b071eefe62a07082fdf4f6fba6a06d4d4d32980d15924851849a9c56b2d79b cleanup: failed to delete container...such container"
Jan 08 07:23:08 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:08.116170523Z" level=error msg="Handler for POST /containers/b9b071eefe62a07082fdf4f6fba6a06d4d4d32980d15924851849a9c56b2d79b/start returned error:...
Jan 08 07:23:08 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:08.783230028Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/16916044f40bf3d64918c3692589ec3d746d2445...false pid=27280
Jan 08 07:23:09 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:09.563702774Z" level=info msg="shim reaped" id=16916044f40bf3d64918c3692589ec3d746d244538d891b1e30d14cb30ccfb51
Jan 08 07:23:09 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:09.570572405Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 08 07:23:09 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:09.572574505Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 08 07:23:09 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:09.853855720Z" level=error msg="16916044f40bf3d64918c3692589ec3d746d244538d891b1e30d14cb30ccfb51 cleanup: failed to delete container...such container"
Jan 08 07:23:09 n1.ha.env.lab.io dockerd[1155]: time="2021-01-08T07:23:09.853938270Z" level=error msg="Handler for POST /containers/16916044f40bf3d64918c3692589ec3d746d244538d891b1e30d14cb30ccfb51/start returned error:...
Hint: Some lines were ellipsized, use -l to show in full.

网上相关原因挺多，有说是cpu飚高导致的，有说是mem不足导致的，prometheus记录了当时节点环境的cpu/mem数据跟正常情况相差无几。

也有说是内核原因导致的，但是这个问题并不是必现是偶现的。

临时解决方法是参考了：https://github.com/moby/moby/issues/40835#issuecomment-730981130