解决神经网络训练时，代码没正常结束导致内存泄露显存不足的问题

本文链接：https://blog.csdn.net/weixin_53610475/article/details/134714016

0.神经网络常见的内存泄露

神经网络使用显卡加速的时候，会将数据移到显存上，很多时候如果程序没有正常结束，显存上的数据没有被清空继续占用着显存空间，再次运行程序就可能会出现显存不足的问题。

1.查看显存占用情况

nvidia-smi查看

Thu Nov 30 15:37:46 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.105.01   Driver Version: 515.105.01   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| N/A   50C    P0    N/A /  N/A |    380MiB /  4096MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3240      G   /usr/lib/xorg/Xorg                162MiB |
|    0   N/A  N/A      3519      G   /usr/bin/gnome-shell               26MiB |
|    0   N/A  N/A      4439      G   /usr/lib/firefox/firefox          141MiB |
|    0   N/A  N/A     10010      G   ...RendererForSitePerProcess       44MiB |
+-----------------------------------------------------------------------------+