【神经网络】深度学习训练时GPU显存不足 & 浮点数在计算机中的存储 & 大小端字节序 & 内存对齐

最新推荐文章于 2024-10-16 15:12:35 发布

大熊の笔记

最新推荐文章于 2024-10-16 15:12:35 发布

阅读量1.4k

点赞数

分类专栏： Linux 文章标签：深度学习神经网络 tensorflow

本文链接：https://blog.csdn.net/kernelxiao/article/details/107459409

版权

本文介绍了在深度学习训练时遇到GPU显存不足的问题及其解决方案，探讨了浮点数在计算机内存中的存储方式，包括大小端字节序和内存对齐的概念，帮助理解数据在内存中的表示和处理方式。

摘要由CSDN通过智能技术生成

问题描述

2020-07-20 10:48:49.753261: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ****************************************___**____***************************************____________
2020-07-20 10:48:49.753329: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cudnn_pooling_gpu.cc:140 : Resource exhausted: OOM when allocating tensor with shape[1098,16,128,128,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1098,16,128,128,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{
   {
   node gradients/max_pooling3d/MaxPool3D_grad/MaxPool3DGrad}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "testCNN.py", line 208, in <module>
    _, mean_loss_val = sess.run([optimizer, mean_loss], feed_dict=train_feed_dict)
  File "/root/caozx/22-anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 958, in run
    run_metadata_ptr)
  File "/root/caozx/22-anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1181, in _run
    feed_dict_tensor, options, run_metadata)
  File "/root/caozx/22-anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/root/caozx/22-anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
    raise type(e

最低0.47元/天解锁文章