1.环境
ubuntu16.04
python3.5
models-1.12.0
2.问题与解决
2.1 问题
tensorflow objection detection运行train.py时,出现如下问题:
Error reported to Coordinator: OOM when allocating tensor with shape[24,96,180,330] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
I0530 02:17:51.322153 140545912301312 tf_logging.py:115] Error reported to Coordinator: OOM when allocating tensor with shape[24,96,180,330] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node FeatureExtractor/MobilenetV2/expanded_conv_1/expand/Conv2D (defined at /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/layers/python/layers/layers.py:1057) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FeatureExtractor/MobilenetV2/expanded_conv/project/BatchNorm/FusedBatchNorm, FeatureExtractor/MobilenetV2/expanded_conv_1/expand/weights/read/_549)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
2.2 解决
不仅在在train.py中指定多张卡:
os.environ['CUDA_VISIBLE_DEVICES']='5,6,7'
config = tf.ConfigProto(allow_soft_placement=True)
config.gpu_options.allow_growth = True
session = tf.Session(config=config)
而且需要做如下修改:
参考:https://blog.csdn.net/jidebingfeng/article/details/81557434
解决(训练时,多加下面两个参数):
train.py --num_clones=3 --ps_tasks=1
参考
1. TensorFlow Object Detection API 多GPU 卡平行计算,加速模型训练速度教学