Faster-RCNN训练问题解决：GPU内存

I1002 16:29:32.222652 27395 layer_factory.hpp:77] Creating layer bbox_pred
I1002 16:29:32.222658 27395 net.cpp:106] Creating Layer bbox_pred
I1002 16:29:32.222661 27395 net.cpp:454] bbox_pred <- fc7_drop7_0_split_1
I1002 16:29:32.222666 27395 net.cpp:411] bbox_pred -> bbox_pred
I1002 16:29:32.223363 27395 net.cpp:150] Setting up bbox_pred
I1002 16:29:32.223371 27395 net.cpp:157] Top shape: 1 8 (8)
I1002 16:29:32.223372 27395 net.cpp:165] Memory required for data: 504340896
I1002 16:29:32.223382 27395 layer_factory.hpp:77] Creating layer loss_cls
I1002 16:29:32.223389 27395 net.cpp:106] Creating Layer loss_cls
I1002 16:29:32.223392 27395 net.cpp:454] loss_cls <- cls_score
I1002 16:29:32.223397 27395 net.cpp:454] loss_cls <- labels
I1002 16:29:32.223402 27395 net.cpp:411] loss_cls -> cls_loss
I1002 16:29:32.223414 27395 layer_factory.hpp:77] Creating layer loss_cls
I1002 16:29:32.223579 27395 net.cpp:150] Setting up loss_cls
I1002 16:29:32.223585 27395 net.cpp:157] Top shape: (1)
I1002 16:29:32.223587 27395 net.cpp:160]     with loss weight 1
I1002 16:29:32.223595 27395 net.cpp:165] Memory required for data: 504340900
I1002 16:29:32.223598 27395 layer_factory.hpp:77] Creating layer loss_bbox
I1002 16:29:32.223606 27395 net.cpp:106] Creating Layer loss_bbox
I1002 16:29:32.223609 27395 net.cpp:454] loss_bbox <- bbox_pred
I1002 16:29:32.223613 27395 net.cpp:454] loss_bbox <- bbox_targets
I1002 16:29:32.223618 27395 net.cpp:454] loss_bbox <- bbox_inside_weights
I1002 16:29:32.223620 27395 net.cpp:454] loss_bbox <- bbox_outside_weights
I1002 16:29:32.223625 27395 net.cpp:411] loss_bbox -> bbox_loss
I1002 16:29:32.223676 27395 net.cpp:150] Setting up loss_bbox
I1002 16:29:32.223680 27395 net.cpp:157] Top shape: (1)
I1002 16:29:32.223682 27395 net.cpp:160]     with loss weight 1
I1002 16:29:32.223686 27395 net.cpp:165] Memory required for data: 504340904
I1002 16:29:32.223690 27395 layer_factory.hpp:77] Creating layer rpn_conv1
I1002 16:29:32.223697 27395 net.cpp:106] Creating Layer rpn_conv1
I1002 16:29:32.223701 27395 net.cpp:454] rpn_conv1 <- conv5_relu5_0_split_1
I1002 16:29:32.223706 27395 net.cpp:411] rpn_conv1 -> rpn_conv1
I1002 16:29:32.228289 27395 net.cpp:150] Setting up rpn_conv1
I1002 16:29:32.228297 27395 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I1002 16:29:32.228299 27395 net.cpp:165] Memory required for data: 509452712
I1002 16:29:32.228318 27395 layer_factory.hpp:77] Creating layer rpn_relu1
I1002 16:29:32.228324 27395 net.cpp:106] Creating Layer rpn_relu1
I1002 16:29:32.228327 27395 net.cpp:454] rpn_relu1 <- rpn_conv1
I1002 16:29:32.228333 27395 net.cpp:397] rpn_relu1 -> rpn_conv1 (in-place)
I1002 16:29:32.228442 27395 net.cpp:150] Setting up rpn_relu1
I1002 16:29:32.228448 27395 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I1002 16:29:32.228451 27395 net.cpp:165] Memory required for data: 514564520
I1002 16:29:32.228453 27395 layer_factory.hpp:77] Creating layer rpn_conv1_rpn_relu1_0_split
I1002 16:29:32.228462 27395 net.cpp:106] Creating Layer rpn_conv1_rpn_relu1_0_split
I1002 16:29:32.228466 27395 net.cpp:454] rpn_conv1_rpn_relu1_0_split <- rpn_conv1
I1002 16:29:32.228469 27395 net.cpp:411] rpn_conv1_rpn_relu1_0_split -> rpn_conv1_rpn_relu1_0_split_0
I1002 16:29:32.228476 27395 net.cpp:411] rpn_conv1_rpn_relu1_0_split -> rpn_conv1_rpn_relu1_0_split_1
I1002 16:29:32.228504 27395 net.cpp:150] Setting up rpn_conv1_rpn_relu1_0_split
I1002 16:29:32.228509 27395 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I1002 16:29:32.228513 27395 net.cpp:157] Top shape: 2 256 39 64 (1277952)
I1002 16:29:32.228515 27395 net.cpp:165] Memory required for data: 524788136
I1002 16:29:32.228518 27395 layer_factory.hpp:77] Creating layer rpn_cls_score
I1002 16:29:32.228526 27395 net.cpp:106] Creating Layer rpn_cls_score
I1002 16:29:32.228529 27395 net.cpp:454] rpn_cls_score <- rpn_conv1_rpn_relu1_0_split_0
I1002 16:29:32.228534 27395 net.cpp:411] rpn_cls_score -> rpn_cls_score
I1002 16:29:32.229225 27395 net.cpp:150] Setting up rpn_cls_score
I1002 16:29:32.229233 27395 net.cpp:157] Top shape: 2 18 39 64 (89856)
I1002 16:29:32.229236 27395 net.cpp:165] Memory required for data: 525147560
I1002 16:29:32.229241 27395 layer_factory.hpp:77] Creating layer rpn_bbox_pred
I1002 16:29:32.229252 27395 net.cpp:106] Creating Layer rpn_bbox_pred
I1002 16:29:32.229256 27395 net.cpp:454] rpn_bbox_pred <- rpn_conv1_rpn_relu1_0_split_1
I1002 16:29:32.229261 27395 net.cpp:411] rpn_bbox_pred -> rpn_bbox_pred
I1002 16:29:32.229975 27395 net.cpp:150] Setting up rpn_bbox_pred
I1002 16:29:32.229982 27395 net.cpp:157] Top shape: 2 36 39 64 (179712)
I1002 16:29:32.229985 27395 net.cpp:165] Memory required for data: 525866408
I1002 16:29:32.229990 27395 layer_factory.hpp:77] Creating layer silence_rpn_cls_score
I1002 16:29:32.229997 27395 net.cpp:106] Creating Layer silence_rpn_cls_score
I1002 16:29:32.230000 27395 net.cpp:454] silence_rpn_cls_score <- rpn_cls_score
I1002 16:29:32.230006 27395 net.cpp:150] Setting up silence_rpn_cls_score
I1002 16:29:32.230008 27395 net.cpp:165] Memory required for data: 525866408
I1002 16:29:32.230010 27395 layer_factory.hpp:77] Creating layer silence_rpn_bbox_pred
I1002 16:29:32.230015 27395 net.cpp:106] Creating Layer silence_rpn_bbox_pred
I1002 16:29:32.230016 27395 net.cpp:454] silence_rpn_bbox_pred <- rpn_bbox_pred
I1002 16:29:32.230020 27395 net.cpp:150] Setting up silence_rpn_bbox_pred
I1002 16:29:32.230023 27395 net.cpp:165] Memory required for data: 525866408
I1002 16:29:32.230026 27395 net.cpp:228] silence_rpn_bbox_pred does not need backward computation.
I1002 16:29:32.230028 27395 net.cpp:228] silence_rpn_cls_score does not need backward computation.
I1002 16:29:32.230031 27395 net.cpp:228] rpn_bbox_pred does not need backward computation.
I1002 16:29:32.230033 27395 net.cpp:228] rpn_cls_score does not need backward computation.
I1002 16:29:32.230036 27395 net.cpp:228] rpn_conv1_rpn_relu1_0_split does not need backward computation.
I1002 16:29:32.230038 27395 net.cpp:228] rpn_relu1 does not need backward computation.
I1002 16:29:32.230041 27395 net.cpp:228] rpn_conv1 does not need backward computation.
I1002 16:29:32.230044 27395 net.cpp:226] loss_bbox needs backward computation.
I1002 16:29:32.230049 27395 net.cpp:226] loss_cls needs backward computation.
I1002 16:29:32.230053 27395 net.cpp:226] bbox_pred needs backward computation.
I1002 16:29:32.230057 27395 net.cpp:226] cls_score needs backward computation.
I1002 16:29:32.230059 27395 net.cpp:226] fc7_drop7_0_split needs backward computation.
I1002 16:29:32.230062 27395 net.cpp:226] drop7 needs backward computation.
I1002 16:29:32.230065 27395 net.cpp:226] relu7 needs backward computation.
I1002 16:29:32.230067 27395 net.cpp:226] fc7 needs backward computation.
I1002 16:29:32.230070 27395 net.cpp:226] drop6 needs backward computation.
I1002 16:29:32.230072 27395 net.cpp:226] relu6 needs backward computation.
I1002 16:29:32.230075 27395 net.cpp:226] fc6 needs backward computation.
I1002 16:29:32.230078 27395 net.cpp:226] roi_pool_conv5 needs backward computation.
I1002 16:29:32.230082 27395 net.cpp:226] conv5_relu5_0_split needs backward computation.
I1002 16:29:32.230084 27395 net.cpp:226] relu5 needs backward computation.
I1002 16:29:32.230087 27395 net.cpp:226] conv5 needs backward computation.
I1002 16:29:32.230089 27395 net.cpp:226] relu4 needs backward computation.
I1002 16:29:32.230092 27395 net.cpp:226] conv4 needs backward computation.
I1002 16:29:32.230094 27395 net.cpp:226] relu3 needs backward computation.
I1002 16:29:32.230096 27395 net.cpp:226] conv3 needs backward computation.
I1002 16:29:32.230099 27395 net.cpp:226] pool2 needs backward computation.
I1002 16:29:32.230103 27395 net.cpp:226] norm2 needs backward computation.
I1002 16:29:32.230104 27395 net.cpp:226] relu2 needs backward computation.
I1002 16:29:32.230108 27395 net.cpp:226] conv2 needs backward computation.
I1002 16:29:32.230109 27395 net.cpp:226] pool1 needs backward computation.
I1002 16:29:32.230113 27395 net.cpp:226] norm1 needs backward computation.
I1002 16:29:32.230114 27395 net.cpp:226] relu1 needs backward computation.
I1002 16:29:32.230118 27395 net.cpp:226] conv1 needs backward computation.
I1002 16:29:32.230123 27395 net.cpp:228] data does not need backward computation.
I1002 16:29:32.230125 27395 net.cpp:270] This network produces output bbox_loss
I1002 16:29:32.230128 27395 net.cpp:270] This network produces output cls_loss
I1002 16:29:32.230146 27395 net.cpp:283] Network initialization done.
I1002 16:29:32.230211 27395 solver.cpp:60] Solver scaffolding done.
Loading pretrained model weights from data/imagenet_models/ZF.v2.caffemodel
I1002 16:29:32.319813 27395 net.cpp:816] Ignoring source layer pool5_spm6
I1002 16:29:32.319828 27395 net.cpp:816] Ignoring source layer pool5_spm6_flatten
I1002 16:29:32.348173 27395 net.cpp:816] Ignoring source layer fc8
I1002 16:29:32.348183 27395 net.cpp:816] Ignoring source layer prob
Solving...
/home/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py:100: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  fg_inds, size=fg_rois_per_this_image, replace=False)
/home/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py:113: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  bg_inds, size=bg_rois_per_this_image, replace=False)
/home/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py:120: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  labels[fg_rois_per_this_image:] = 0
/home/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py:176: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  bbox_targets[ind, start:end] = bbox_target_data[ind, 1:]
/home/py-faster-rcnn/tools/../lib/roi_data_layer/minibatch.py:177: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  bbox_inside_weights[ind, start:end] = cfg.TRAIN.BBOX_INSIDE_WEIGHTS
F1002 16:29:32.513561 27395 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***