报错原因及解决方案
报错现象:
Allocator (GPU_0_bfc) ran out of memory trying to allocate 200.00MiB (rounded to 209715200). Current allocation summary follows.
<省略>
Resource exhausted: OOM when allocating tensor with shape[51200,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc.
<省略>
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[51200,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node word/word/bi-lstm-0/bi-lstm-0/bw/bw/while/lstm_cell/MatMul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[add/_77]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[51200,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node word/word/bi-lstm-0/bi-lstm-0/bw/bw/while/lstm_cell/MatMul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
报错原因在于GPU显存不够用了,可以在运行过程通过命令watch -n 1 nvidia-smi
查看GPU的显存利用率,退出用ctrl+C
。注意中间那一列才是显存利用情况"已分配显存/可用总显存",最右边那列是GPU利用率,这是另一回事(这就和内存利用率和CPU利用率是两回事是同一个道理)。
解决方案有几种,根据自己的情况来选择:
- 如果没有设置允许tensorflow根据需要自动增加申请的显存,可以先尝试这个,看看用上单个GPU的所有显存能否解决。设置自动增长的代码见后文。
- 如果网络中用到了RNN,可以使用
swap_memory=True
选项以减少其对显存的占用。例如tf.nn.bidirectional_dynamic_rnn()
方法就有这个参数。设置之后,tensorflow会将RNN前向运算产生但反向传播需要用到的tensor从GPU转移到CPU中(从显存转移到内存),这几乎不会(甚至完全不会)带来性能上的损失。 - 减小batch_size或减小RNN的序列的最大长度(即时间步长)
- 换个显存更大的GPU吧
P.S. 设置显存自动增长的代码:
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)