Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals.

An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection.

We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features

INTRODUCTION

Fast R-CNN , achieves near real-time rates using very deep networks , when ignoring the time spent on region proposals.

computing proposals with a deep convolutional neural network

On top of these convolutional features, we construct an RPN by adding a few additional convolutional layers that simultaneously regress region bounds and objectness scores at each location on a regular grid. The RPN is thus a kind of fully convolutional network (FCN) and can be trained end-toend specifically for the task for generating detection proposals.

we introduce novel “anchor” boxes that serve as references at multiple scales and aspect ratios. Our scheme can be thought of as a pyramid of regression references (Figure 1, c), which avoids enumerating images or filters of multiple scales or aspect ratios.

we propose a training scheme that alternates between fine-tuning for the region proposal task and then fine-tuning for object detection, while keeping the proposals fixed.

Our method is not only a cost-efficient solution for practical usage, but also an effective way of improving object detection accuracy.

RELATED WORK

Object Proposals.

Deep Networks for Object Detection.

FASTER R-CNN

Our object detection system, called Faster R-CNN, is composed of two modules.

1. The first module is a deep fully convolutional network that proposes regions
2. The second module is the Fast R-CNN detector that uses the proposed regions.

Region Proposal Networks

![RPN][faster_rcnn_figure3]

A Region Proposal Network (RPN) takes animage (of any size) as input and outputs a set of rectangular object proposals, each with an objectness score.

To generate region proposals, we slide a small network over the convolutional feature map output by the last shared convolutional layer. This small network takes as input an $n\times n$ spatial window of the input convolutional feature map. Each sliding window is mapped to a lower-dimensional feature (256-d for ZF and 512-d for VGG, with ReLU following).

This feature is fed into two sibling fullyconnected layers—a box-regression layer (reg) and a box-classification layer (cls). We use n = 3 in this paper, noting that the effective receptive field on the input image is large (171 and 228 pixels for ZF and VGG, respectively).

This mini-network is illustrated at a single position in Figure 3 (left). Note that because the mini-network operates in a sliding-window fashion, the fully-connected layers are shared across all spatial locations. This architecture is naturally implemented with an n × n convolutional layer followed by two sibling 1 × 1 convolutional layers (for $reg$ and $cls$, respectively).

Anchors

maximum possible proposals for each location is denoted as $k$.

the $reg$ layer has $4k$ outputs encoding the coordinates of $k$ boxes, and the $cls$ layer outputs $2k$ scores that estimate probability of object or not object for each proposal.

Translation-Invariant Anchors
An important property of our approach is that it is translation invariant

Multi-Scale Anchors as Regression References

1. The first way is based on image/feature pyramids. The images are resized at multiple scales, and feature maps (HOG [8] or deep convolutional features [9], [1], [2]) are computed for each scale
2. The second way is to use sliding windows of multiple scales (and/or aspect ratios) on the feature maps. Different aspect ratios are trained separately using different filter sizes
3. Our method classifies and regresses bounding boxes with reference to anchor boxes of multiple scales and aspect ratios.

Loss Function

We assign a positive label to two kinds of anchors:

1. the anchor/anchors with the highest Intersection-overUnion (IoU) overlap with a ground-truth box
2. an anchor that has an IoU overlap higher than 0.7 with any ground-truth box.

We assign a negative label to a non-positive anchor if its IoU ratio is lower than 0.3 for all ground-truth boxes.

loss function for an image:
$$L(p_i,t_i)=\frac{1}{N_{cls}}\sum _i{L}_{cls}(P_i,P_i^{\ast })+\lambda \frac{1}{N_{reg}}\sum _i{p}_i^{\ast }{L}_{reg}(t_i,t_i^{\ast })$$

The term $p_i^{\ast }{L}_{reg}$ means the regression loss is activated only for positive anchors ($p_i^{\ast }=1$) and is disabled otherwise ($p_i^{\ast }=0$).

In our formulation, the features used for regression are of the same spatial size (3 × 3) on the feature maps. To account for varying sizes, a set of k bounding-box regressors are learned. Each regressor is responsible for one scale and one aspect ratio, and the k regressors do not share weights.

Training RPNs

Sharing Features for RPN and Fast R-CNN

We discuss three ways for training networks with features shared:

1. Alternating training. In this solution, we first train RPN, and use the proposals to train Fast R-CNN. The network tuned by Fast R-CNN is then used to initialize RPN, and this process is iterated.
2. Approximate joint training. The backward propagation takes place as usual, where for the shared layers the backward propagated signals from both the RPN loss and the Fast R-CNN loss are combined.
3. Non-approximate joint training. a theoretically valid backpropagation solver should also involve gradients w.r.t. the box coordinates. we need an RoI pooling layer that is differentiable w.r.t. the box coordinates. This is a nontrivial problem and a solution can be given by an “RoI warping” layer

4-Step Alternating Training.

1. In the first step, we train the RPN as described. This network is initialized with an ImageNet-pre-trained model and fine-tuned end-to-end for the region proposal task.
2. In the second step, we train a separate detection network by Fast R-CNN using the proposals generated by the step-1 RPN. This detection network is also initialized by the ImageNet-pre-trained model. At this point the two networks do not share convolutional layers.
3. In the third step, we use the detector network to initialize RPN training, but we fix the shared convolutional layers and only fine-tune the layers unique to RPN. Now the two networks share convolutional layers.
4. Finally, keeping the shared convolutional layers fixed, we fine-tune the unique layers of Fast R-CNN.

Implementation Details

We note that our algorithm allows predictions that are larger than the underlying receptive field.

To reduce redundancy, we adopt non-maximum suppression (NMS) on the proposal regions based on their cls scores.

EXPERIMENTS

Experiments on PASCAL VOC

Ablation Experiments on RPN.

NMS does not harm the detection mAP and may reduce false alarms.

the cls scores account for the accuracy of the highest ranked proposals.

This suggests that the highquality proposals are mainly due to the regressed box bounds. The anchor boxes, though having multiple scales and aspect ratios, are not sufficient for accurate detection.

it suggests that the proposal quality of RPN+VGG is better than that of RPN+ZF.

Performance of VGG-16.

Sensitivities to Hyper-parameters.

using anchors of multiple sizes as the regression references is an effective solution.

scales and aspect ratios are not disentangled dimensions for the detection accuracy.

the result is insensitive to λ in a wide range.

Analysis of Recall-to-IoU.

It is more appropriate to use this metric to diagnose the proposal method than to evaluate it.

RPN method behaves gracefully

One-Stage Detection vs. Two-Stage Proposal + Detection.

OverFeat is a one-stage, class-specific detection pipeline, and ours is a two-stage cascade consisting of class-agnostic proposals and class-specific detections.

In OverFeat, the region-wise features come from a sliding window of one aspect ratio over a scale pyramid. These features are used to simultaneously determine the location and category of objects.

In RPN, the features are from square (3×3) sliding windows and predict proposals relative to anchors with different scales and aspect ratios.

Experiments on MS COCO

Faster R-CNN in ILSVRC & COCO 2015 competitions

From MS COCO to PASCAL VOC

CONCLUSION