# SSD: Single Shot MultiBox Detector

Wei Liu,Dragomir Anguelov ,Dumitru Erhan ,Christian Szegedy ,Scott Reed ,Cheng-Yang Fu & Alexander C. Berg

• UNC Chapel Hill
• Zoox Inc
• University of Michigan, Ann-Arbor

## Abstract

We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. SSD is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets confirm that SSD has competitive accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For 300 × 300 input, SSD achieves 74.3% mAP1 on VOC2007 test at 59 FPS on a Nvidia Titan X and for 512 × 512 input, SSD achieves 76.9% mAP, outperforming a comparable state-of-the-art Faster R-CNN model. Compared to other single stage methods, SSD has much better accuracy even with a smaller input image size. Code is available at: https://github.com/weiliu89/caffe/tree/ssd .

## 1 Introduction

Current state-of-the-art object detection systems are variants of the following approach: hypothesize bounding boxes, resample pixels or features for each box, and apply a high-quality classifier. This pipeline has prevailed on detection benchmarks since the Selective Search work [1] through the current leading results on PASCAL VOC, COCO, and ILSVRC detection all based on Faster R-CNN[2] albeit with deeper features such as [3]. While accurate, these approaches have been too computationally intensive for embedded systems and, even with high-end hardware, too slow for real-time applications.Often detection speed for these approaches is measured in seconds per frame (SPF), and even the fastest high-accuracy detector, Faster R-CNN, operates at only 7 frames per second (FPS). There have been many attempts to build faster detectors by attacking each stage of the detection pipeline (see related work in Sec. 4), but so far, significantly increased speed comes only at the cost of significantly decreased detection accuracy.

This paper presents the first deep network based object detector that does not resample pixels or features for bounding box hypotheses and and is as accurate as approaches that do. This results in a significant improvement in speed for high-accuracy detection (59 FPS with mAP 74.3% on VOC2007 test, vs. Faster R-CNN 7 FPS with mAP 73.2% or YOLO 45 FPS with mAP 63.4%). The fundamental improvement in speed comes from eliminating bounding box proposals and the subsequent pixel or feature resampling stage. We are not the first to do this (cf [4,5]), but by adding a series of improvements, we manage to increase the accuracy significantly over previous attempts. Our improvements include using a small convolutional filter to predict object categories and offsets in bounding box locations, using separate predictors (filters) for different aspect ratio detections, and applying these filters to multiple feature maps from the later stages of a network in order to perform detection at multiple scales. With these modifications—especially using multiple layers for prediction at different scales—we can achieve high-accuracy using relatively low resolution input, further increasing detection speed. While these contributions may seem small independently, we note that the resulting system improves accuracy on real-time detection for PASCAL VOC from 63.4% mAP for YOLO to 74.3% mAP for our SSD. This is a larger relative improvement in detection accuracy than that from the recent, very high-profile work on residual networks [3]. Furthermore, significantly improving the speed of high-quality detection can broaden the range of settings where computer vision is useful.

We summarize our contributions as follows:

• 我们推出了SSD，这是一种适用于多种类别的单发检测器，它比以前的最新单发检测器（YOLO）速度更快，并且精度更高，实际上与执行显式区域建议和池化(包括faster R-CNN)的较慢技术一样准确。

We introduce SSD, a single-shot detector for multiple categories that is faster than the previous state-of-the-art for single shot detectors (YOLO), and significantly more accurate, in fact as accurate as slower techniques that perform explicit region proposals and pooling (including Faster R-CNN).

• SSD的核心是使用应用于特征图的小型卷积滤波器，为一组固定的默认边界框预测类别得分和框偏移。

The core of SSD is predicting category scores and box offsets for a fixed set of default bounding boxes using small convolutional filters applied to feature maps.

• 为了获得较高的检测精度，我们从不同比例的特征图生成不同比例的预测，并按宽高比将显示明确分开。

To achieve high detection accuracy we produce predictions of different scales from feature maps of different scales, and explicitly separate predictions by aspect ratio.

• 这些设计功能即使在低分辨率输入图像上也可以实现简单的端到端训练和高精度，从而进一步提高了速度与精度之间的权衡。

These design features lead to simple end-to-end training and high accuracy, even on low resolution input images, further improving the speed vs accuracy trade-off.

• 实验包括对通过PASCAL VOC，COCO和ILSVRC评估的具有可变输入大小的模型进行时序和精度分析，并将其与一系列最新技术进行了比较。

Experiments include timing and accuracy analysis on models with varying input size evaluated on PASCAL VOC, COCO, and ILSVRC and are compared to a range of recent state-of-the-art approaches.

## 2 The Single Shot Detector (SSD)

This section describes our proposed SSD framework for detection (Sec. 2.1) and the associated training methodology (Sec. 2.2). Afterwards, Sec. 3 presents dataset-specific model details and experimental results.

Fig. 1: SSD framework. (a) SSD only needs an input image and ground truth boxes for each object during training. In a convolutional fashion, we evaluate a small set (e.g. 4) of default boxes of different aspect ratios at each location in several feature maps with different scales (e.g. 8 × 8 and 4 × 4 in (b) and ©). For each default box, we predict both the shape offsets and the confidences for all object categories ( ( c 1 , c 2 , . . . , c p ) (c_1, c_2, ...,c_p) ). At training time, we first match these default boxes to the ground truth boxes. For example, we have matched two default boxes with the cat and one with the dog, which are treated as positives and the rest as negatives. The model loss is a weighted sum between localization loss (e.g. Smooth L1 [6]) and confidence loss (e.g. Softmax).

### 2.1 Model

SSD方法基于前馈卷积网络，该网络产生固定大小的边界框集合，并为这些框中存在的对象类实例打分，然后进行非最大抑制步骤以产生最终检测结果。 早期的网络层基于用于高质量图像分类的标准体系结构（在任何分类层之前均被截断），我们将其称为基础网络2。 然后，我们将辅助结构添加到网络，以产生具有以下关键特征的检测结果：

The SSD approach is based on a feed-forward convolutional network that produces a fixed-size collection of bounding boxes and scores for the presence of object class instances in those boxes, followed by a non-maximum suppression step to produce the final detections. The early network layers are based on a standard architecture used for high quality image classification (truncated before any classification layers), which we will call the base network2. We then add auxiliary structure to the network to produce detections with the following key features:

Multi-scale feature maps for detection We add convolutional feature layers to the end of the truncated base network. These layers decrease in size progressively and allow predictions of detections at multiple scales. The convolutional model for predicting detections is different for each feature layer (cf Overfeat[4] and YOLO[5] that operate on a single scale feature map).

Fig. 2: A comparison between two single shot detection models: SSD and YOLO [5]. Our SSD model adds several feature layers to the end of a base network, which predict the offsets to default boxes of different scales and aspect ratios and their associated confidences. SSD with a 300 × 300 input size significantly outperforms its 448 × 448 YOLO counterpart in accuracy on VOC2007 test while also improving the speed.

Convolutional predictors for detection Each added feature layer (or optionally an existing feature layer from the base network) can produce a fixed set of detection predictions using a set of convolutional filters. These are indicated on top of the SSD network architecture in Fig. 2. For a feature layer of size m × n m × n with p channels, the basic element for predicting parameters of a potential detection is a 3 × 3 × p 3 × 3 × p small kernel that produces either a score for a category, or a shape offset relative to the default box coordinates. At each of the m × n m × n locations where the kernel is applied, it produces an output value. The bounding box offset output values are measured relative to a default box position relative to each feature map location (cf the architecture of YOLO[5] that uses an intermediate fully connected layer instead of a convolutional filter for this step).

Default boxes and aspect ratios We associate a set of default bounding boxes with each feature map cell, for multiple feature maps at the top of the network. The default boxes tile the feature map in a convolutional manner, so that the position of each box relative to its corresponding cell is fixed. At each feature map cell, we predict the offsets relative to the default box shapes in the cell, as well as the per-class scores that indicate the presence of a class instance in each of those boxes. Specifically, for each box out of k at a given location, we compute c class scores and the 4 offsets relative to the original default box shape. This results in a total of ( c + 4 ) k (c + 4)k filters that are applied around each location in the feature map, yielding ( c + 4 ) k m n (c + 4)kmn outputs for a m × n m × n feature map. For an illustration of default boxes, please refer to Fig. 1. Our default boxes are similar to the anchor boxes used in Faster R-CNN [2], however we apply them to several feature maps of different resolutions. Allowing different default box shapes in several feature maps let us efficiently discretize the space of possible output box shapes.

### 2.2 Training

The key difference between training SSD and training a typical detector that uses region proposals, is that ground truth information needs to be assigned to specific outputs in the fixed set of detector outputs. Some version of this is also required for training in YOLO[5] and for the region proposal stage of Faster R-CNN[2] and MultiBox[7]. Once this assignment is determined, the loss function and back propagation are applied end-to-end. Training also involves choosing the set of default boxes and scales for detection as well as the hard negative mining and data augmentation strategies.

Matching strategy During training we need to determine which default boxes correspond to a ground truth detection and train the network accordingly. For each ground truth box we are selecting from default boxes that vary over location, aspect ratio, and scale. We begin by matching each ground truth box to the default box with the best jaccard overlap (as in MultiBox [7]). Unlike MultiBox, we then match default boxes to any ground truth with jaccard overlap higher than a threshold (0.5). This simplifies the learning problem, allowing the network to predict high scores for multiple overlapping default boxes rather than requiring it to pick only the one with maximum overlap.

Training objective The SSD training objective is derived from the MultiBox objective [7,8] but is extended to handle multiple object categories. Let x i j p = 1 , 0 x^p_{ij} = {1,0} be an indicator for matching the j-th default box to the j-th ground truth box of category p. In the matching strategy above, we can have ∑ i x i j p ≥ 1 \sum_ix^p_{ij} \ge 1 . The overall objective loss function is a weighted sum of the localization loss (loc) and the confidence loss (conf):

L ( x , c , l , g ) = 1 N ( L c o n f ( x , c ) + α L l o c ( x , l , g ) ) (1) \tag{1} L(x,c,l,g)=\frac{1}{N}(L_{conf}(x,c)+\alpha L_{loc}(x,l,g))

where N is the number of matched default boxes. If N = 0 N = 0 , wet set the loss to 0. The localization loss is a Smooth L1 loss [6] between the predicted box ( l ) (l) and the ground truth box ( g ) (g) parameters. Similar to Faster R-CNN [2], we regress to offsets for the center ( c x , c y ) (cx,cy) of the default bounding box ( d ) (d) and for its width ( w ) (w) and height ( h ) (h) .

L l o c ( x , l , g ) = ∑ i ∈ P o s N ∑ m ∈ c x , c y , w , h x i j k s m o o t h L 1 ( l i m − g ^ j m ) L_{loc}(x,l,g)=\displaystyle\sum_{i\in Pos}^N \displaystyle\sum_{m\in {cx,cy,w,h}}x^k_{ij}smooth_{L1}(l_i^m-\hat{g}^m_j)

g ^ j c x = ( g j c x − d i c x ) / d i w         g ^ j c y = ( g j c y − d i c y ) / d i h (2) \tag{2} \hat{g}^{cx}_j =(g^{cx}_j-d^{cx}_i)/d^w_i \space \space\space\space\space\space\space \hat{g}^{cy}_j=(g^{cy}_j-d^{cy}_i)/d_i^h

g ^ j w = l o g ( g j w d i w )         g ^ j h = l o g ( g j h d i h ) \hat{g}^w_j=log(\frac{g^w_j}{d^w_i}) \space \space\space\space\space\space\space \hat{g}^h_j=log(\frac{g^h_j}{d_i^h})

The confidence loss is the softmax loss over multiple classes confidences ©.

L c o n f ( x , c ) = − ∑ i ∈ P o s N x i j p l o g ( c ^ i p ) − ∑ i ∈ N e g l o g ( ) c i 0 ^    w h e r e    c ^ i p = e x p ( c i p ) ∑ p e x p ( c i p ) (3) \tag{3} L_{conf}(x,c)=-\displaystyle\sum_{i\in Pos}^N x^p_{ij}log(\hat{c}^p_i)-\displaystyle\sum_{i\in Neg}log()\hat{c^0_i} \space\space where \space\space \hat{c}^p_i=\frac{exp(c^p_i)}{\sum_p exp(c^p_i)}

and the weight term α α is set to 1 by cross validation.

07-17

09-19 9万+
08-11 1838
07-06 1264
06-05 9815
12-31