Hint: If you want to see a list of allocated tensors when OOM happens

I have seen OOMs happen several epochs into training in tensorflow, my best guess is that if your model is at the borderline of using all the GPU memory then internal memory allocation issues such as memory fragmentation or the how temporary RAM is being used can make the model OOM even if it was able to train on a few epochs. The only thing to do is reduce RAM usage by reducing the batch size or using a smaller model and trying again.

简单地说，就是GPU使用率太高了，然后进行新的运算时就崩溃了。查看GPU利用率的方法：

点击打开链接

深度学习中最直观的方式就是减小batch_size或者hidden_layer中的单元数