tensorflow objection detection多张GPU卡 (Allocator (GPU_0_bfc) ran out of memory)

1.环境

ubuntu16.04
python3.5
models-1.12.0

2.问题与解决

2.1 问题

tensorflow objection detection运行train.py时，出现如下问题：

Error reported to Coordinator: OOM when allocating tensor with shape[24,96,180,330] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

I0530 02:17:51.322153 140545912301312 tf_logging.py:115] Error reported to Coordinator: OOM when allocating tensor with shape[24,96,180,330] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node FeatureExtractor/MobilenetV2/expanded_conv_1/expand/Conv2D (defined at /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/layers/python/layers/layers.py:1057)  = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FeatureExtractor/MobilenetV2/expanded_conv/project/BatchNorm/FusedBatchNorm, FeatureExtractor/MobilenetV2/expanded_conv_1/expand/weights/read/_549)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

2.2 解决

不仅在在train.py中指定多张卡：

os.environ['CUDA_VISIBLE_DEVICES']='5,6,7'
config = tf.ConfigProto(allow_soft_placement=True)
config.gpu_options.allow_growth = True
session = tf.Session(config=config)

而且需要做如下修改： 

参考：https://blog.csdn.net/jidebingfeng/article/details/81557434

解决（训练时，多加下面两个参数）：

train.py --num_clones=3  --ps_tasks=1  

参考

1. TensorFlow Object Detection API 多GPU 卡平行计算，加速模型训练速度教学