在公司生产环境中在一次重启kubelet的时候,启动失败
报错日志查看:journalctl -xefu kubelet
注释:这个命令是查看服务启动失败的命令。
查看报错日志:
Jan 05 09:47:57 paasmir2 kubelet[55193]: E0105 01:47:57.630086 55221 raw.go:140] Failed to watch directory “/sys/fs/cgroup/memory/system.slice/dev-mqueue.mount”: inotify_add_watch /sys/fs/cgroup/memory/system.slice/dev-mqueue.mount: no space left on device
Jan 05 09:47:57 paasmir2 kubelet[55193]: E0105 01:47:57.630129 55221 raw.go:140] Failed to watch directory “/sys/fs/cgroup/memory/system.slice”: inotify_add_watch /sys/fs/cgroup/memory/system.slice/dev-mqueue.mount: no space left on device
Jan 05 09:47:57 paasmir2 kubelet[55193]: F0105 01:47:57.630140 55221 kubelet.go:1344] Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/memory/system.slice/dev-mqueue.mount: no space left on device
解决方法
执行cat /proc/sys/fs/inotify/max_user_watches检查结果,如果是8192,需要执行如下命令:
echo 1048576 > /proc/sys/fs/inotify/max_user_watches