Kubernetes 1.18 Warning ImageGCFailed Error failed to get imageFs info: unable to find data in memor

Initially, Kubernetes came up properly. But when we running the same pod multiple times for testing Kubelet get restarted every 5s.

When I checked,

Kubectl get events

7m43s       Normal    Starting                  node/rajasvm   Starting kubelet.
7m43s       Normal    NodeHasSufficientMemory   node/rajasvm   Node rajasvm status is now: NodeHasSufficientMemory
7m43s       Normal    NodeHasNoDiskPressure     node/rajasvm   Node rajasvm status is now: NodeHasNoDiskPressure
7m43s       Normal    NodeHasSufficientPID      node/rajasvm   Node rajasvm status is now: NodeHasSufficientPID
7m26s       Normal    Starting                  node/rajasvm   Starting kubelet.
7m26s       Normal    NodeHasSufficientMemory   node/rajasvm   Node rajasvm status is now: NodeHasSufficientMemory
7m26s       Normal    NodeHasNoDiskPressure     node/rajasvm   Node rajasvm status is now: NodeHasNoDiskPressure
7m26s       Normal    NodeHasSufficientPID      node/rajasvm   Node rajasvm status is now: NodeHasSufficientPID
7m9s        Normal    Starting                  node/rajasvm   Starting kubelet.
7m9s        Warning   ImageGCFailed             node/rajasvm   failed to get imageFs info: unable to find data in memory cache
6m52s       Normal    Starting                  node/rajasvm   Starting kubelet.
6m35s       Normal    Starting                  node/rajasvm   Starting kubelet.
6m35s       Normal    NodeHasSufficientMemory   node/rajasvm   Node rajasvm status is now: NodeHasSufficientMemory
6m35s       Normal    NodeHasNoDiskPressure     node/rajasvm   Node rajasvm status is now: NodeHasNoDiskPressure
6m35s       Normal    NodeHasSufficientPID      node/rajasvm   Node rajasvm status is now: NodeHasSufficientPID

处理方法
这个问题明显是Docker ImageGC 的问题。应该是内存不够了。导致的kubelet挂掉了。但是查看df -h 。磁盘空间还是够的。

journalct 查看kubelet日志
journalctl -u kubelet | grep -i garbage

解决方案。先让docker自己清理一下镜像文件。然后重启kubelet和docker
 

May 27 17:00:05 rajasvm kubelet[20241]: E0527 17:00:05.374190   20241 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
May 27 17:00:22 rajasvm kubelet[20401]: E0527 17:00:22.152485   20401 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
May 27 17:00:39 rajasvm kubelet[20548]: E0527 17:00:39.141443   20548 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
May 27 17:00:55 rajasvm kubelet[20693]: E0527 17:00:55.953994   20693 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
May 27 17:01:12 rajasvm kubelet[20848]: E0527 17:01:12.668267   20848 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
May 27 17:01:29 rajasvm kubelet[20994]: E0527 17:01:29.676793   20994 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
May 27 17:01:46 rajasvm kubelet[21136]: E0527 17:01:46.367956   21136 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
May 27 17:02:03 rajasvm kubelet[21282]: E0527 17:02:03.181850   21282 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
May 27 17:02:03 rajasvm kubelet[21282]: E0527 17:02:03.181865   21282 kubelet.go:1301] Image garbage collection failed multiple times in a row: failed to get imageFs info: unable to find data in memory cache

I got the solution, it looks like deleted docker images somehow didn't clean up properly.No idea about this. But below solution worked for me

docker system prune
systemctl stop kubelet
systemctl stop docker
systemctl start docker
systemctl start kubelet