RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory

坚持写博客，分享自己的在学习、工作中的所得

1. 给自己做备忘
2. 对知识点记录、总结，加深理解
3. 给有需要的人一些帮助，少踩一个坑，多走几步路

尽量以合适的方式排版，图文兼有
如果写的有误，或者有不理解的，均可在评论区留言
如果内容对你有帮助，欢迎点赞 👍 收藏 ⭐留言 📝。
虽然平台并不会有任何奖励，但是我会很开心，可以让我保持写博客的热情

---

昨天安装完tensorflow之后，没有问题，也跑了代码训练模型。今天突然就不行了，电脑没关过机。
使用tf报了以下错误

RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory

import tensorflow as tf
print(tf.__version__)
print(tf.test.is_built_with_cuda())
print(tf.test.is_gpu_available())

2.4.0
True
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-27-b9f4dadd08f2> in <module>
      2 print(tf.__version__)
      3 print(tf.test.is_built_with_cuda())
----> 4 print(tf.test.is_gpu_available())

~/miniconda3/envs/py38/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
    338               'in a future version' if date is None else ('after %s' % date),
    339               instructions)
--> 340       return func(*args, **kwargs)
    341 
    342     doc_controls.set_deprecated(new_func)

~/miniconda3/envs/py38/lib/python3.8/site-packages/tensorflow/python/framework/test_util.py in is_gpu_available(***failed resolving arguments***)
   1583 
   1584   try:
-> 1585     for local_device in device_lib.list_local_devices():
   1586       if local_device.device_type == "GPU":
   1587         gpu_info = gpu_util.compute_capability_from_device_desc(local_device)

~/miniconda3/envs/py38/lib/python3.8/site-packages/tensorflow/python/client/device_lib.py in list_local_devices(session_config)
     41     serialized_config = session_config.SerializeToString()
     42   return [
---> 43       _convert(s) for s in _pywrap_device_lib.list_devices(serialized_config)
     44   ]

RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory

nvidia-smi

Wed Feb 24 14:32:18 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.32.00    Driver Version: 455.32.00    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:65:00.0  On |                  N/A |
| 34%   32C    P8    20W / 350W |  24240MiB / 24265MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1140      G   /usr/lib/xorg/Xorg                 35MiB |
|    0   N/A  N/A      1695      G   /usr/lib/xorg/Xorg                106MiB |
|    0   N/A  N/A      1843      G   /usr/bin/gnome-shell               25MiB |
|    0   N/A  N/A      2720      G   /usr/lib/firefox/firefox            4MiB |
|    0   N/A  N/A      6667      G   /usr/lib/firefox/firefox            4MiB |
|    0   N/A  N/A      8566      C   ...nda3/envs/py38/bin/python    23535MiB |
|    0   N/A  N/A      9446      G   /usr/lib/firefox/firefox            4MiB |
|    0   N/A  N/A      9746      G   /usr/lib/firefox/firefox            4MiB |
|    0   N/A  N/A     29957      C   ...nda3/envs/py38/bin/python      499MiB |
+-----------------------------------------------------------------------------+

可以看到3090整整24G显存全被占用了
从上面可以看到占用最多的PID是8566

kill -9 8566

将进程杀死之后占用的现存就释放了

---

如果内容对你有帮助，或者觉得写的不错
🏳️‍🌈欢迎点赞 👍 收藏 ⭐留言 📝
有问题，请在评论区留言