我们基本上将模型组件分为 5 种类型。
- 骨干:通常是一个 FCN 网络来提取特征图,例如 ResNet、MobileNet。
- 颈部:骨干和头部之间的组件,例如FPN、PAFPN。
- head:特定任务的组件,例如 bbox 预测和掩码预测。
- roiextractor:从特征图中提取RoI特征的部分,例如RoI Align。
- loss:用于计算损失的 head 组件,例如FocalLoss、L1Loss 和 GHMLoss。
开发新组件
添加新的主干
这里我们以 MobileNet 为例展示如何开发新组件。
1**.定义一个新的主干(例如MobileNet)**
创建一个新文件mmdet/models/backbones/mobilenet.py。
import torch.nn as nn
from ..builder import BACKBONES
@BACKBONES.register_module()
class MobileNet(nn.Module):
def __init__(self, arg1, arg2):
pass
def forward(self, x): # should return a tuple
pass
- 导入模块
您可以将以下行添加到 mmdet/models/backbones/init.py
from .mobilenet import MobileNet
或者添加
custom_imports = dict(
imports=['mmdet.models.backbones.mobilenet'],
allow_failed_imports=False)
到配置文件以避免修改原始代码。
3. 在你的配置文件中使用主干
model = dict(
…
backbone=dict(
type=‘MobileNet’,
arg1=xxx,
arg2=xxx),
…
添加新的脖子
1. 定义颈部(例如 PAFPN)
创建一个新文件mmdet/models/necks/pafpn.py。
from ..builder import NECKS
@NECKS.register_module()
class PAFPN(nn.Module):
def __init__(self,
in_channels,
out_channels,
num_outs,
start_level=0,
end_level=-1,
add_extra_convs=False):
pass
def forward(self, inputs):
# implementation is ignored
pass
- 导入模块
您可以将以下行添加到mmdet/models/necks/init.py,
from .pafpn import PAFPN
或者添加
custom_imports = dict(
imports=['mmdet.models.necks.pafpn.py'],
allow_failed_imports=False)
到配置文件,避免修改原始代码。
3.修改配置文件
neck=dict(
type='PAFPN',
in_channels=[256, 512, 1024, 2048],
out_channels=256,
num_outs=5)
添加新的头
这里我们以双头 R-CNN为例展示如何开发一个新的头部,如下所示。
首先,在mmdet/models/roi_heads/bbox_heads/double_bbox_head.py. 双头 R-CNN 实现了一个新的 bbox 头用于对象检测。要实现一个bbox head,基本上我们需要实现新模块的三个功能如下。
from mmdet.models.builder import HEADS
from .bbox_head import BBoxHead
@HEADS.register_module()
class DoubleConvFCBBoxHead(BBoxHead):
r"""Bbox head used in Double-Head R-CNN
/-> cls
/-> shared convs ->
\-> reg
roi features
/-> cls
\-> shared fc ->
\-> reg
""" # noqa: W605
def __init__(self,
num_convs=0,
num_fcs=0,
conv_out_channels=1024,
fc_out_channels=1024,
conv_cfg=None,
norm_cfg=dict(type='BN'),
**kwargs):
kwargs.setdefault('with_avg_pool', True)
super(DoubleConvFCBBoxHead, self).__init__(**kwargs)
def forward(self, x_cls, x_reg):
其次,如有必要,实施新的 RoI Head。我们计划DoubleHeadRoIHead从StandardRoIHead. 我们可以发现aStandardRoIHead已经实现了以下功能。
import torch
from mmdet.core import bbox2result, bbox2roi, build_assigner, build_sampler
from ..builder import HEADS, build_head, build_roi_extractor
from .base_roi_head import BaseRoIHead
from .test_mixins import BBoxTestMixin, MaskTestMixin
@HEADS.register_module()
class StandardRoIHead(BaseRoIHead, BBoxTestMixin, MaskTestMixin):
"""Simplest base roi head including one bbox head and one mask head.
"""
def init_assigner_sampler(self):
def init_bbox_head(self, bbox_roi_extractor, bbox_head):
def init_mask_head(self, mask_roi_extractor, mask_head):
def forward_dummy(self, x, proposals):
def forward_train(self,
x,
img_metas,
proposal_list,
gt_bboxes,
gt_labels,
gt_bboxes_ignore=None,
gt_masks=None):
def _bbox_forward(self, x, rois):
def _bbox_forward_train(self, x, sampling_results, gt_bboxes, gt_labels,
img_metas):
def _mask_forward_train(self, x, sampling_results, bbox_feats, gt_masks,
img_metas):
def _mask_forward(self, x, rois=None, pos_inds=None, bbox_feats=None):
def simple_test(self,
x,
proposal_list,
img_metas,
proposals=None,
rescale=False):
"""Test without augmentation."""
Double Head 的修改主要是在 bbox_forward 逻辑中,它继承了StandardRoIHead. 在 中mmdet/models/roi_heads/double_roi_head.py,我们实现了新的 RoI Head,如下所示:
from ..builder import HEADS
from .standard_roi_head import StandardRoIHead
@HEADS.register_module()
class DoubleHeadRoIHead(StandardRoIHead):
"""RoI head for Double Head RCNN
https://arxiv.org/abs/1904.06493
"""
def __init__(self, reg_roi_scale_factor, **kwargs):
super(DoubleHeadRoIHead, self).__init__(**kwargs)
self.reg_roi_scale_factor = reg_roi_scale_factor
def _bbox_forward(self, x, rois):
bbox_cls_feats = self.bbox_roi_extractor(
x[:self.bbox_roi_extractor.num_inputs], rois)
bbox_reg_feats = self.bbox_roi_extractor(
x[:self.bbox_roi_extractor.num_inputs],
rois,
roi_scale_factor=self.reg_roi_scale_factor)
if self.with_shared_head:
bbox_cls_feats = self.shared_head(bbox_cls_feats)
bbox_reg_feats = self.shared_head(bbox_reg_feats)
cls_score, bbox_pred = self.bbox_head(bbox_cls_feats, bbox_reg_feats)
bbox_results = dict(
cls_score=cls_score,
bbox_pred=bbox_pred,
bbox_feats=bbox_cls_feats)
return bbox_results
最后,用户需要添加模块 mmdet/models/bbox_heads/init.py,mmdet/models/roi_heads/init.py从而相应的注册表可以找到并加载它们。
或者,用户可以添加
custom_imports=dict(
imports=['mmdet.models.roi_heads.double_roi_head', 'mmdet.models.bbox_heads.double_bbox_head'])
到配置文件并实现相同的目标。
双头R-CNN的配置文件如下
_base_ = '../faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py'
model = dict(
roi_head=dict(
type='DoubleHeadRoIHead',
reg_roi_scale_factor=1.3,
bbox_head=dict(
_delete_=True,
type='DoubleConvFCBBoxHead',
num_convs=4,
num_fcs=2,
in_channels=256,
conv_out_channels=1024,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=80,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0., 0., 0., 0.],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=False,
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=False, loss_weight=2.0),
loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=2.0))))
从MMDetection 2.0开始,配置系统支持继承配置,让用户可以专注于修改。双头R-CNN主要使用了一个新的DoubleHeadRoIHead和一个新的 DoubleConvFCBBoxHead,参数根据__init__每个模块的功能设置。
添加新的损失
假设您想MyLoss为边界框回归添加一个新的损失为。要添加新的损失函数,用户需要在mmdet/models/losses/my_loss.py. 装饰weighted_loss器使损失能够为每个元素加权。
import torch
import torch.nn as nn
from ..builder import LOSSES
from .utils import weighted_loss
@weighted_loss
def my_loss(pred, target):
assert pred.size() == target.size() and target.numel() > 0
loss = torch.abs(pred - target)
return loss
@LOSSES.register_module()
class MyLoss(nn.Module):
def __init__(self, reduction='mean', loss_weight=1.0):
super(MyLoss, self).__init__()
self.reduction = reduction
self.loss_weight = loss_weight
def forward(self,
pred,
target,
weight=None,
avg_factor=None,
reduction_override=None):
assert reduction_override in (None, 'none', 'mean', 'sum')
reduction = (
reduction_override if reduction_override else self.reduction)
loss_bbox = self.loss_weight * my_loss(
pred, target, weight, reduction=reduction, avg_factor=avg_factor)
return loss_bbox
然后用户需要将它添加到mmdet/models/losses/init.py.
from .my_loss import MyLoss, my_loss
或者,您可以添加
custom_imports=dict(
imports=['mmdet.models.losses.my_loss'])
到配置文件并实现相同的目标。
要使用它,请修改该loss_xxx字段。由于 MyLoss 是用于回归的,所以需要修改loss_bboxhead 中的字段。
loss_bbox=dict(type='MyLoss', loss_weight=1.0))