SSD系列算法的主要文章
- SSD:Single Shot MultiBox Detector
- DSSD: Deconvolutional Single Shot Detector
- FSSD:Feature Fusion Single Shot Multibox Detector
- RefineDet: Single-Shot Refinement Neural Network for Object Detection
- RfbNet:Receptive Field Block Net for Accurate and Fast Object Detection
- M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network
- Pelee:Pelee: A Real-Time Object Detection System on Mobile Devices
算法简介
- state-of-the-art object detection systems
- SSD
SSD算法是一种直接预测目标类别和bounding box的多目标检测算法。与faster rcnn相比,该算法没有生成 proposal 的过程,这极大的提高了检测速度。针对不同大小的目标检测,传统的做法是先将图像转换成不同大小(图像金字塔),然后分别检测,最后将结果综合起来(NMS)。而SSD算法则利用不同卷积层的 feature map 进行综合达到同样的效果。算法的主干网络是VGG16,但是将最后两个全连接层改成卷积层,并额外又增加了4个卷积层来构造网络结构。对其中5种不同的卷积层的输出(feature map)分别用两个不同的 3×3 的卷积核进行卷积,一个输出分类用的confidence,每个default box 生成21个类别(voc dataset)confidence;一个输出回归用的 localization,每个 default box 生成4个坐标值(x, y, w, h)。此外,这5个feature map还经过 PriorBox 层生成 prior box(生成的是坐标)。上述5个feature map中每一层的default box的数量是给定的(8732个)。最后将前面三个计算结果分别合并然后传给loss层。
Default box
-
feature map cell
feature map cell 是指feature map中每一个小格子,如下图Fig 1中8x8 feature map和4x4 feature map分别有64和16个cell。 -
default box
default box 是指在feature map的每个小格(cell)上都有一系列固定大小的box。假设每个feature map cell有k个default box,那么对于每个default box都需要预测c个类别score和4个offset,那么如果一个feature map的大小是m×n,也就是有m*n个feature map cell,那么这个feature map就一共有(c+4)k * mn 个输出。这些输出个数的含义是:采用3×3的卷积核对该层的feature map卷积时卷积核的个数,包含两部分(实际code是分别用不同数量的33卷积核对该层feature map进行卷积):数量**ckmn是confidence输出**,表示每个default box的confidence,也就是类别的概率;数量4km*n是localization输出,表示每个default box回归后的坐标)。 -
prior box
prior box,是指实际中选择的default box(每一个feature map cell 不是k个default box都取)。也就是说default box是一种概念,prior box是实际的选取。训练中一张完整的图片送进网络获得各个feature map,对于正样本训练来说,需要先将prior box与ground truth box做匹配,匹配成功说明这个prior box所包含的是个目标,但离完整目标的ground truth box还有段距离,训练的目的是保证default box的分类confidence的同时将prior box尽可能回归到ground truth box。 举个列子:假设一个训练样本中有2个ground truth box,所有的feature map中获取的prior box一共有8732个。那个可能分别有10、20个prior box能分别与这2个ground truth box匹配上。训练的损失包含定位损失和回归损失两部分。
作者的实验表明default box的shape数量越多,效果越好。且default box 是应用在多个不同层的feature map上的。 -
边界框的location(cx, cy, w, h)
The bounding box offset output values are measured relative to a default box position relative to each feature map location。这4个值其实是真实坐标的变换值,用来对每个feature map位置的默认框位置进行测量。 -
Choosing scales and aspect ratios for default boxes
假设我们用m个feature maps做预测,那么对于每个featuer map而言其default box的scale是按以下公式计算的:
s k = s m i n + s m a x − s m i n m − 1 ( k − 1 ) , k ∈ [ 1 , m ] s_k = s_{min} + \frac {s_{max} - s_{min}} {m - 1} (k - 1), k \in [1, m] sk=smin+m−1smax−smin(k−1),k∈[1,m]
where s m i n s_{min} smin是0.2,表示the lowest layer的scale是0.2; s m a x s_{max} smax是0.9,表示the highest layer的scale是0.9。至于aspect ratio,用 a r a_r ar表示为 a r ∈ 1 , 2 , 3 , 1 2 , 1 3 a_r \in{1, 2, 3, \frac {1} {2}, \frac {1} {3} } ar∈1,2,3,21,31:注意这里一共有5种aspect ratio。
则每个default box的宽计算公式为: w k a = s k ( a r ) w_{k}^a=s_k \sqrt(a_r) wka=sk(ar),高的计算公式为 h k a = s k / ( a r ) h_k^a= {s_k} /{\sqrt(a_r)} hka=sk/(ar),对于aspect ratio为1时,default box的scale计算式为 s k ′ = ( s k s k + 1 ) s_k^{'} =\sqrt(s_ks_{k+1}) sk′=(sksk+1),对于每个feature map cell一共有6种default box。
由此可以得出 default box在不同的feature层有不同的scale,在同一个feature层又有不同的aspect ratio,如下表所示.因此基本上可以覆盖输入图像中的各种形状和大小的object。 最后得到的prior box个数为:(38x38x4 + 19x19x6 + 10x10x6 + 5x5x6 + 3x3x4 + 1x1x4)= 8732 。
feature map | feature map size | mi_size | max_size | aspct ration | step | offset | variance |
---|---|---|---|---|---|---|---|
conv4_3 | 38x38 | 30 | 60 | 1,2 | 8 | 0.5 | 0.1,0.2 |
fc6 | 19x19 | 60 | 111 | 1,2,3 | 16 | 0.5 | 0.1,0.2 |
conv6_2 | 10x10 | 111 | 162 | 1,2,3 | 32 | 0.5 | 0.1,0.2 |
conv7_2 | 5x5 | 111 | 213 | 1,2,3 | 64 | 0.5 | 0.1,0.2 |
conv8_2 | 3x3 | 213 | 264 | 1,2 | 100 0.5 | 0.1,0.2 | |
conv9_2 | 1x1 | 264 | 315 | 1,2 | 300 | 0.5 | 0.1,0.2 |
正负样本的选取
将prior box 和 grount truth box 按照IOU(JaccardOverlap)进行匹配,匹配成功则这个prior box就是positive example(正样本),如果匹配不上,就是negative example(负样本),显然这样产生的负样本的数量要远远多于正样本。这里将前向loss进行排序,选择最高的num_sel个prior box序号集合 ?。那么如果Match成功后的正样本序号集合?。那么最后正样本集为 ?−?∩?,负样本集为 ?−? ∩?。同时可以通过规范num_sel的数量(是正样本数量的三倍)来控制使得最后正、负样本的比例在 1:3 左右。
数据增强
对每一张image进行如下之一变换获取一个batch进行训练:
- 直接使用原始的图像(即不进行变换)
- 采样一个patch,保证与GT之间最小的IoU为:0.1,0.3,0.5,0.7 或 0.9
- 完全随机的采样一个patch
VOC数据集制作
Introduction
The main goal of this challenge is to recognize objects from a number of visual object classes in realistic scenes (i.e. not pre-segmented objects). It is fundamentally a supervised learning learning problem in that a training set of labelled images is provided. The twenty object classes that have been selected are:
- Person: person
- Animal: bird, cat, cow, dog, horse, sheep
- Vehicle: aeroplane, bicycle, boat, bus, car, motorbike, train
- Indoor: bottle, chair, dining table, potted plant, sofa, tv/monitor
20 classes. The train/val data has 11,530 images containing 27,450 ROI annotated objects and 6,929 segmentations.
There are three main object recognition competitions: classification, detection, and segmentation, a competition on action classification, and a competition on large scale recognition run by ImageNet. In addition there is a “taster” competition on person layout.
Classification/Detection Competitions
- Classification: For each of the twenty classes, predicting presence/absence of an example of that class in the test image.
- Detection: Predicting the bounding box and label of each object from the twenty target classes in the test image.
Segmentation Competition
- Segmentation: Generating pixel-wise segmentations giving the class of the object visible at each pixel, or “background” otherwise.
Action Classification Competition
-
Action Classification: Predicting the action(s) being performed by a person in a still image.
-
VOC2012
- Annotations
- 2008_007420.xml
- ImageSets
- Action
- Layout
- Main
- Segmentation
- JPEGImages
- SegmentationClass
- SegmentationObject
- Annotations
Annotations:中主要存放xml文件,每一个xml对应一张图像,并且每个xml中存放的是标记的各个目标的位置和类别信息,命名通常与对应的原始图像一样
JPEGImages:自己的原始图像放在JPEGImages文件夹
ImageSets:
---- Action 预测静态图像中人做出的动作(running、jumping等等)
---- Layout 即人体轮廓布局。该任务的目标是预测人体部位(head、hand、feet等等)的bounding box和对应的label
---- Main 存放的是目标识别的数据,总共分为20类,主要有xxx_test.txt , xxx_train.txt, xxx_val.txt,xxx_trainval.txt四个文件,前面的表示图像的name,后面的1代表正样本,-1代表负样本。如
tail -n5 person_train.txt
2011_003253 -1
2011_003255 1
2011_003259 1
2011_003274 -1
2011_003276 -1
---- Segmentation 存放分割的数据,train.txt中存放的是训练集的图片编号,val.txt中存放的是验证集的图片编号,trainval是上面两者的合并集合.
VOC2012/ImageSets/Main/train.txt 保存了所有训练集的文件名,从 VOC2012/JPEGImages/ 存放的是所有的原图片,而VOC2012/Annotations/ 则是原图片对应的标签文件。
Annotations
Annotations文件夹中存放的是xml格式的标签文件,每一个xml文件都对应于JPEGImages文件夹中的一张图片。
xml的文件格式如下所示:
<annotation>
<filename>2008_000156.jpg</filename> // 文件名
<folder>VOC2012</folder>
<object> // 检测到到物体信息
<name>car</name> // 物体类别
<actions>
<jumping>0</jumping>
<other>0</other>
<phoning>1</phoning>
<playinginstrument>0</playinginstrument>
<reading>0</reading>
<ridingbike>0</ridingbike>
<ridinghorse>0</ridinghorse>
<running>0</running>
<takingphoto>0</takingphoto>
<usingcomputer>0</usingcomputer>
<walking>0</walking>
</actions>
<bndbox>
<xmax>63</xmax>
<xmin>1</xmin>
<ymax>375</ymax>
<ymin>84</ymin>
</bndbox>
<difficult>0</difficult> // 目标是否难以识别,0表示容易识别
<pose>Unspecified</pose> // 物体的姿态
<point>
<x>26</x>
<y>183</y>
</point>
</object>
<segmented>0</segmented> // 是否用于分割
<size> // 图像大小
<depth>3</depth>
<height>375</height>
<width>500</width>
</size>
<source> // 图片来源
<annotation>PASCAL VOC2012</annotation>
<database>The VOC2012 Database</database>
<image>flickr</image>
</source>
</annotation>
ImageSets
ImageSets存放的是每一种类型的challenge对应的图像数据。在ImageSets下有四个文件夹:
- Action下存放的是人的动作(例如running、jumping等等,这也是VOC challenge的一部分)
- Layout下存放的是具有人体部位的数据(人的head、hand、feet等等,这也是VOC challenge的一部分)
- Main下存放的是图像物体识别的数据,总共分为20类。
- Segmentation下存放的是可用于分割的数据。
Main文件夹下包含了20个分类的***_train.txt、***_val.txt和***_trainval.txt。这些txt中的内容都差不多。前面的表示图像的name,后面的1代表正样本,-1代表负样本。_train中存放的是训练使用的数据,每一个class的train数据都有5717个。_val中存放的是验证结果使用的数据,每一个class的val数据都有5823个。_trainval将上面两个进行了合并,每一个class有11540个。需要保证的是train和val两者没有交集,也就是训练数据和验证数据不能有重复,在选取训练数据的时候 ,也应该是随机产生的。
JPEGImages
JPEGImages文件夹中包含了PASCAL VOC所提供的所有的图片,包含训练图片和测试图片,共有17125张。这些图像都是以“年份_编号.jpg”格式命名的。图片的像素尺寸大小不一,但是横向图的尺寸大约在500375左右,纵向图的尺寸大约在375500左右,基本不会偏差超过100。在之后的训练中,第一步就是将这些图片都resize到300300或是500500,所有原始图片不能离这个标准过远。这些图像就是用来进行训练和测试验证的图像数据。
SegmentationClass
含了2913张图片,每一张图片都对应JPEGImages里面的相应编号的图片,图片的像素颜色共有20种,对应20类物体。
SegmentationObject
包含了2913张图片,图片编号都与Class里面的图片编号相同。这里面的图片和Class里面图片的区别在于,这是针对Object的。在Class里面,一张图片里如果有多架飞机,那么会全部标注为红色。而在Object里面,同一张图片里面的飞机会被不同颜色标注出来。
制作 VOC2012数据集
VOC数据集的制作过程主要包括以下几步:
- 数据准备,将要制作成voc格式数据集的数据进行处理生成label_list.txt,其中label_list.txt格式如下所示:
filename targets (tx, ty, bx, by) classes
car/00001.jpg 10, 30, 120, 90 20
- 根据VOC数据格式生成符合VOC数据集格式要求的文件,对于目标检测主要有Annotations/.xml、ImageSets/main/.txt、JPEGImages/*.jpg。
Step1 生成VOC2012对应目录
VOC2012目录如下图所示:
-- VOC2012
|-- Annotations
|-- ImageSets
| |-- Action
| |-- Layout
| |-- Main
| `-- Segmentation
|-- JPEGImages
|-- SegmentationClass
`-- SegmentationObject
对应代码为:
# -*- coding: utf-8 -*-
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import os, sys
def make_voc2012_directory():
os.makedirs('VOC2012/Annotations')
os.makedirs('VOC2012/ImageSets')
os.makedirs('VOC2012/ImageSets/Action')
os.makedirs('VOC2012/ImageSets/Main')
os.makedirs('VOC2012/ImageSets/Layout')
os.makedirs('VOC2012/ImageSets/Segmentation')
os.makedirs('VOC2012/JPEGImages')
os.makedirs('VOC2012/SegmentationClass')
os.makedirs('VOC2012/SegmentationObject')
if __name__ == '__main__':
make_voc2012_directory()
Step2 在Annotations目录下生成图片对应的XML文件
生成xml文件有2种策略:
- 全量生成,从头开始生成
- 增量生成 ,在已有的基础上生成
全量生成Annotations目录下的XML文件
# -*- coding: utf-8 -*-
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import os, sys
import cv2
import numpy as np
from lxml.etree import Element, SubElement, tostring
from xml.dom.minidom import parseString
def full_generate_annotations_xml(root_dir, label_list, Annotations_dir, JPEGImages_dir):
label_list = os.path.join(root_dir, label_list)
i = 1
with open(label_list, 'r') as f:
lines = f.readlines()
print(f'to generating annotations xml file numbers:{len(lines)}')
for line in lines:
line_info = line.rstrip('\n').split(' ')
imgname = os.path.join(root_dir, line_info[0])
img = cv2.imread(imgname)
height, width, channel = img.shape
image_name = '%09d' % i + '.jpg'
# save JPEGImages
new_image_name = JPEGImages_dir + '/%09d' % i + '.jpg'
# construct annotation
node_root = Element('annotation')
node_folder = SubElement(node_root, 'folder')
node_folder.text = 'JPEGImages'
# add image name
node_filename = SubElement(node_root, 'filename')
node_filename.text = image_name
# add image width height channel
node_size = SubElement(node_root, 'size')
node_depth = SubElement(node_size, 'depth')
node_depth.text = '%s' % channel
node_height = SubElement(node_size, 'height')
node_height.text = '%s' % height
node_width = SubElement(node_size, 'width')
node_width.text = '%s' % width
# to be write xml
write_infile = False
# bbox info, include classes
line_info = [int(b) for b in line_info[1:]]
# convert to numpy array,exclude classes
array=np.array(line_info[:-1])
# reshape to Mnx4
bboxs = array.reshape(-1, 4)
for bbox in bboxs:
x, y, w, h = [int(b) for b in bbox]
# filter images, not must
if w < 48 or h < 48:
continue
write_infile=True
left, top, right, bottom = x, y, x + w, y + h
node_object = SubElement(node_root, 'object')
node_name = SubElement(node_object, 'name')
node_name.text = 'car'
node_difficult = SubElement(node_object, 'difficult')
node_difficult.text = '0'
node_bndbox = SubElement(node_object, 'bndbox')
node_xmin = SubElement(node_bndbox, 'xmin')
node_xmin.text = '%s' % left
node_ymin = SubElement(node_bndbox, 'ymin')
node_ymin.text = '%s' % top
node_xmax = SubElement(node_bndbox, 'xmax')
node_xmax.text = '%s' % right
node_ymax = SubElement(node_bndbox, 'ymax')
node_ymax.text = '%s' % bottom
if write_infile:
xml = tostring(node_root, pretty_print=True)
dom = parseString(xml)
# save_xml
save_xml = os.path.join(Annotations_dir, image_name.replace('jpg', 'xml'))
with open(save_xml, 'wb') as f:
f.write(xml)
cv2.imwrite(new_image_name, img)
i = i + 1
print(f'***** full_generate_annotations_xml Done, total numbers:{i} *******************')
if __name__ == '__main__':
# dataset to convert
root_dir = '/path/dataset'
label_list='dataset_label_list.txt'
# Voc dataset directory
Annotations_dir='VOC2012/Annotations'
JPEGImages_dir='VOC2012/JPEGImages'
full_generate_annotations_xml(root_dir, label_list, Annotations_dir, JPEGImages_dir)
增量生成Annotations目录下的XML文件
# -*- coding: utf-8 -*-
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import os, sys
import cv2
import numpy as np
from lxml.etree import Element, SubElement, tostring
from xml.dom.minidom import parseString
def inc_generate_annotations_xml(root_dir, label_list, Annotations_dir, JPEGImages_dir):
label_list = os.path.join(root_dir, label_list)
start_idx = 1
Annotations_xmls = os.listdir(Annotations_dir)
print(f'there are already {len(Annotations_xmls)} Annotations xmls')
start_idx = len(Annotations_xmls)
with open(label_list, 'r') as f:
lines = f.readlines()
print(f'inc_generate_annotations_xml read lines:{len(lines)}')
for line in lines:
line_info = line.rstrip('\n').split(' ')
imgname = os.path.join(root_dir, line_info[0])
img = cv2.imread(imgname)
height, width, channel = img.shape
image_name = '%09d' % start_idx + '.jpg'
# save JPEGImages
new_image_name = JPEGImages_dir + '/%09d' % start_idx + '.jpg'
# construct annotation
node_root = Element('annotation')
node_folder = SubElement(node_root, 'folder')
node_folder.text = 'JPEGImages'
# image name
node_filename = SubElement(node_root, 'filename')
node_filename.text = image_name
# image width height channel
node_size = SubElement(node_root, 'size')
node_depth = SubElement(node_size, 'depth')
node_depth.text = '%s' % channel
node_height = SubElement(node_size, 'height')
node_height.text = '%s' % height
node_width = SubElement(node_size, 'width')
node_width.text = '%s' % width
write_infile = False
# bbounding box info
line_info = [int(b) for b in line_info[1:]]
array=numpy.array(line_info[:-1])
bboxs = array.reshape(-1, 4)
for bbox in bboxs:
x, y, w, h = [int(b) for b in bbox]
# image filter
if w < 48 or h < 48:
continue
write_infile=True
left, top, right, bottom = x, y, x + w, y + h
node_object = SubElement(node_root, 'object')
node_name = SubElement(node_object, 'name')
node_name.text = 'person'
node_difficult = SubElement(node_object, 'difficult')
node_difficult.text = '0'
node_bndbox = SubElement(node_object, 'bndbox')
node_xmin = SubElement(node_bndbox, 'xmin')
node_xmin.text = '%s' % left
node_ymin = SubElement(node_bndbox, 'ymin')
node_ymin.text = '%s' % top
node_xmax = SubElement(node_bndbox, 'xmax')
node_xmax.text = '%s' % right
node_ymax = SubElement(node_bndbox, 'ymax')
node_ymax.text = '%s' % bottom
if write_infile:
xml = tostring(node_root, pretty_print=True)
dom = parseString(xml)
save_xml = os.path.join(Annotations_dir, image_name.replace('jpg', 'xml'))
with open(save_xml, 'wb') as f:
f.write(xml)
cv2.imwrite(new_image_name, img)
start_idx += 1
print(f'***** inc_generate_annotations_xml Done, total numbers:{start_idx} *******************')
if __name__ == '__main__':
# dataset to convert
root_dir = '/path/dataset'
label_list='dataset_label_list.txt'
# Voc dataset directory
Annotations_dir='VOC2012/Annotations'
JPEGImages_dir='VOC2012/JPEGImages'
inc_generate_annotations_xml(root_dir, label_list, Annotations_dir, JPEGImages_dir)
转换后的图片验证
这里可以通过读取voc格式的转换后的数据进行验证,看是转换是否成功,具体操作如下:
# -*- coding: utf-8 -*-
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import os, sys
import cv2
import numpy as np
import matplotlib.pyplot as plt
import xml.etree.ElementTree as ET
%matplotlib inline
classes = ["__background__","aeroplane", "bicycle", "bird", "boat", "bottle",
"bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant",
"sheep", "sofa", "train", "tvmonitor"]
class_to_ind = dict(zip(classes, range(len(classes))))
def AnnotationTransform(xml_path):
xml_file = open(xml_path, 'r')
# xml
tree=ET.parse(xml_file)
# targets
root = tree.getroot()
# Transforms a VOC annotation into a Tensor of bbox coords and label index
res = np.empty((0, 5))
for obj in root.iter('object'):
difficult = obj.find('difficult').text
name = obj.find('name').text.lower().strip()
if name not in classes or int(difficult) == 1:
continue
bbox = obj.find('bndbox')
pts = ['xmin', 'ymin', 'xmax', 'ymax']
bndbox = []
for i, pt in enumerate(pts):
cur_pt = int(bbox.find(pt).text) - 1
bndbox.append(cur_pt)
label_idx = class_to_ind[name]
bndbox.append(label_idx)
res = np.vstack((res, bndbox))
return res
if __name__ == '__main__':
JPEGImages_dir = 'VOC2012/JPEGImages'
Annotations_dir = 'VOC2012/Annotations'
result =[]
JPEGImages = os.listdir(JPEGImages_dir)
for img in JPEGImages:
imgname = os.path.join(JPEGImages_dir, img)
img = cv2.imread(imgname)
annotation_xml = os.path.join(Annotations_dir, img.replace('jpg', 'xml'))
bboxs = AnnotationTransform(annotation_xml)
for bbox in bboxs:
tx, ty, bx, by, label = [int(b) for b in bbox]
cv2.rectangle(img, (tx, ty), (bx, by), (255, 0, 255), 1)
result.append([img, label])
idx = 1
for k in range(len(result)):
try:
plt.ion()
plt.figure(idx)
plt.title(result[k][1])
plt.imshow(result[k][0][:,:,::-1])
except Exception as e:
print(e)
finally:
idx+=1
Step3 在Main目录下生成对应的数据集的txt文件
这一步主要是对数据集进行分割生成训练集、测试集和验证集并保储成txt文件。可以根据数据集命名.
# -*- coding: utf-8 -*-
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import os, sys
import random
def make_imageset_label(ImageSets_Main_dir, Annotations_dir, label_list):
total_xml = os.listdir(Annotations_dir)
total_xml_list = range(len(total_xml))
label_list_path = os.path.join(ImageSets_Main_dir, label_list)
with open(label_list_path, 'w') as f:
for i in total_xml_list:
name = total_xml[i][:-4] + '\n'
f.write(name)
print('******************* make_imageset_label Done *******************')
if __name__ == '__main__':
ImageSets_Main_dir='VOC2012/ImageSets/Main'
Annotations_dir='VOC2012/Annotations'
label_list ='trainval.txt'
make_imageset_label(ImageSets_Main_dir, Annotations_dir, label_list)