错误信息
Resource ex hausted: OOM when allocating tensor with shape[200,256,28,28] and****
这是一种调参时常遇到的问题,由于电脑显存不够而导致,我的电脑显存是8g,在调整参数 IMAGES_PER_GPU = 2时,会导致这样的错误,将其改回1错误消失(降低了batch size的大小)。
解决办法
- 减少Batch 的大小
- 分析错误的位置,在哪一层出现显卡不够,比如在全连接层出现的,则降低全连接层的维度,把2048改成1024啥的
- 增加pool 层,降低整个网络的维度。
- 修改输入图片的大小
- 有钱任性,换台电脑
# python train.py --logdir myLog --batch_size 256 --dropout_rate 0.5
OP_REQUIRES failed at conv_ops.cc:636 : Resource exhausted: OOM when allocating tensor with shape[32,32,417,417] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
callbacks=[logging, checkpoint])
File "D:\Anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "D:\Anaconda3\lib\site-packages\keras\engine\training.py", line 1415, in fit_generator
initial_epoch=initial_epoch)
File "D:\Anaconda3\lib\site-packages\keras\engine\training_generator.py", line 213, in fit_generator
class_weight=class_weight)
File "D:\Anaconda3\lib\site-packages\keras\engine\training.py", line 1215, in train_on_batch
outputs = self.train_function(ins)
File "D:\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2666, in __call__
return self._call(inputs)
File "D:\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2636, in _call
fetched = self._callable_fn(*array_vals)
File "D:\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1382, in __call__
run_metadata_ptr)
File "D:\Anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 519, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,32,417,417]