pod容器发生OOM错误(by quqi99)

作者：张华 发表于：2020-11-26
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

POD event里看见了OOM, 一个python3.8进程被杀.

"Memory cgroup out of memory: Killed process 1983936 (python3.8) total-vm:66687756kB, anon-rss:16539352kB, file-rss:3116kB, shmem-rss:0kB, UID:1000 pgtables:49344kB oom_score_adj:969"

下列命令用于查询这个POD内的所有进程ps中的RSS内存使用量，它告诉我们这个python3.8命令是Tensorflow, 它用了大量的内存。

$ for i in $(cat sys/fs/cgroup/memory/kubepods/burstable/podcc8b7b27-8ca0-43fd-a47a-0c7461dfce32/aa1e40ad129935970fb9b222a43ba5123556e2e77e1e5af82dc7f176d01b7794/cgroup.procs ); do grep $i ps ; done | awk '{sum+= $6;print $2 " " $6/1024 "MB " substr($0, index($0, $11))} END {print "total memory usage in the cgroup:" sum/1048576 "GB"}' | sort -n -k 2
total memory usage in the cgroup:44.1916GB
2042940 0.671875MB tini -g -- start.sh
2043024 2.10547MB /usr/sbin/nslcd
2042988 2.46094MB /usr/sbin/sshd
2042997 2.55859MB /usr/sbin/nscd
2043388 3.00781MB /bin/bash -l
2042982 3.22266MB sudo -E -H -u rpudd997 PATH=/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/rstudio-server/bin XDG_CACHE_HOME=~/.cache PYTHONPATH= jupyter labhub --ip 0.0.0.0 --config /usr/local/share/jupyter/jupyter_notebook_config.py
2086106 5.30078MB /bin/bash -l
2043055 109.82MB /opt/conda/bin/python /opt/conda/bin/jupyter-labhub --ip 0.0.0.0 --config /usr/local/share/jupyter/jupyter_notebook_config.py
31781 9686.55MB ~/.conda/envs/dsp/bin/python3.8 -m ipykernel_launcher -f ~/.local/share/jupyter/runtime/kernel-3c377b72-7333-4ace-866a-e5882849622a.json
5973 35436.4MB /datasets/misc/DLshare/home/rpudd997/.conda/envs/dsp/bin/python3.8 -m ipykernel_launcher -f ~/.local/share/jupyter/runtime/kernel-5f018973-2d35-4966-8030-7df82a63f621.json

同时，下列命令可以用来查询这个POD下的一些进程的内存限制，显示limit与实际Tensorflow所用的内存差不多(RSS内存是不包括kernel所用内存的)

$cat sys/fs/cgroup/memory/kubepods/burstable/podcc8b7b27-8ca0-43fd-a47a-0c7461dfce32/aa1e40ad129935970fb9b222a43ba5123556e2e77e1e5af82dc7f176d01b7794/memory.limit_in_bytes