CVPR2020 Harmonizing Transferability and Discriminability for Adapting Object Detector

Harmonizing Transferability and Discriminability for Adapting Object Detector

code:https://github.com/chaoqichen/HTCN

Abstract

论文中设计了三个模块，用于源域和目标域之间的对齐，同时能够使得检测器能够对源域特征和目标域特征具有一定的辨别能力。

(1) Importance Weighted Adversarial Training with input Interpolation (IWAT-I), which strengthens the global discriminability by re-weighting the interpolated image-level features; (2) Context-aware Instance-Level Alignment (CILA) module, which enhances the local discriminability by capturing the underlying complementary effect between the instance-level feature and the global context information for the instance-level feature alignment; (3) local feature masks that calibrate the local transferability to provide semantic guidance for the following discriminative pattern alignment.

Introduction

由于源域与目标域之间存在domain shift，在源域训练得到的目标检测器不能够很好地泛化到目标域上。

This hinders the deployment of models in real-world situations where data distributions typically vary from one domain to another. Unsupervised Domain Adaptation (UDA) serves as a promising solution to solve this problem by transferring knowledge from a labeled source domain to a fully unlabeled target domain.

之前的方法对于域自适应中的目标检测问题，通过将对抗学习的方法纳入到传统的目标检测框架中，通过对各层次的图像特征进行对齐，实现模型的可迁移性。作者认为通过对抗的方式学习得到transferability的代价是使得模型的discriminability有所下降。

文章中提到的transferability discriminability解释如下：

Note that, in this paper, the transferability refers to the invariance of the learned representations across domains, and discriminability refers to the ability of the detector to localize and distinguish different instances.

Related Work

Unsupervised Domain Adaptation

Typically, UDA methods propose to bridge different domains by matching the high-order statistics of source and target feature distributions in the latent space.

With insights from the practice of Generative Adversarial Nets (GAN), tremendous works have been done by leveraging the two-player game to achieve domain confusion with Gradient Reversal Layer (GRL) for feature alignment.

In addition, other GAN-based works aim to achieve pixel-level adaptation in virtue of image-toimage translation techniques

UDA for Object Detection

Nevertheless, all these UDA methods do not properly handle the the potential contradiction between transferability and discriminability when adapting object detectors in the context of adversarial adaptation.

Hierarchical Transferability Calibration Network (HTCN)

由于transferability discriminability之间存在一定的矛盾关系，因此作者主要从两个方面入手解决此问题。

(1)通过对多层特征进行匹配对迁移性进行校准（2）将多层级间的特征进行传递，实现跨域的对齐。

Importance Weighted Adversarial Training with Input Interpolation

首先利用cycleGAN对源域和目标域图像进行插值(interpolation)，由于输入不同的样本器包含的可迁移信息是不同的，因此通过计算不同样本所具有的不确定性，计算每个样本对应的重加权系数，来表征每个样本的迁移能力。

IWAT-I模块中，对于域判别器$D_2$的输出为$d_i=D_2\left(G_1\circ G_2(x_i)\right)$,其对应的输入样本$x_i$的域标签信息，其不确定都计算方式如下：
$$v_i=H(d_i)=-d_i\cdot \log (d_i)-(1-d_i)\cdot \log (1-d_i)(1)$$
则每个样本的权重为$1+v_i$，

Images with high uncertainty (hard-to-distinguish by $D_2$) should be up-weighted, vice versa.

该样本re-weighted得到
$$g_i=f_i\times \left(1+v_i\right)(2)$$
所以$D_3$的对抗损失为
$${\mathcal{L}}_{ga}=\mathbb{E}[\log (D_3(G_3(g_i^s)))]+\mathbb{E}[1-\log (D_3(G_3(g_i^t)))](3)$$

Context-Aware Instance-Level Alignment

之前实例级别的对齐是基于ROI-Pooling后的特征进行局部特征的对齐，主要是对域间样本object scale、viewpoint、deformation、appearance特征的对齐。然而存在的问题是这些特征只考虑到目标周围局部特征的对齐，没有考虑整体上下文的一个信息。因此对于不同域之间实例特征是有差别的，二上下文向量通过从底层特征进行融合，能够保证域间的不变性，作者通过将二者融合，实现二者之间的互补。

从backbone不同层获得context vector$f_c^i(i=1,2,3)$,第$i$个图像经过ROI-Polling后得到第$j$个区域对应的特征为$f_{ins}^{i,j}$，通过将其concatenate进行实例特征和context vector的融合得到$[f_c^1,f_c^2,f_c^3,f_{ins}]$，但是此种方法存在一个问题，context vector与实例级的特征之间是独立的，无法实现上述所说的二者的互补。由于这两个特征是不对称的，

采用下述方式将$f_c^i(i=1,2,3)$,与$f_{ins}$相乘，会产生维度爆炸。
$${\mathbf{f}}_{fus}=[f_c^1,f_c^2,f_c^3]\otimes f_{ins}(4)$$
作者利用随机化的方式对特征向量积进行无偏估计，

we propose to leverage the randomized methods as an unbiased estimator of the tensor product.

$${\mathbf{f}}_{fus}=\frac{1}{\sqrt{d}}({\mathbf{R}}_1{\mathbf{f}}_c)\odot ({\mathbf{R}}_2{\mathbf{f}}_{ins})(5)$$

其中$\odot$表示Hadamard product

CA-ILA损失为
$${\mathcal{L}}_{ins}=-\frac{1}{N_s}\sum _{i=1}^{N_s}\sum _{i,j}\log (D_{ins}({\mathbf{f}}_{fus}^{i,j}{)}_s)-\frac{1}{N_t}\sum _{i=1}^{N_t}\sum _{i,j}\log (1-D_{ins}({\mathbf{f}}_{fus}^{i,j}{)}_t)(6)$$

Local Feature Mask for Semantic Consistency

对于不同域的图像尽管scene layouts、object co-occurrence、background之间存在一定的差异，但是对于同一类目标器在不同的域中应该具有相同的sketch，作者假设图像中某些区域相对于其他区域更具有一定的描述性和优势，因此作者在浅层特征的基础上对局部特征计算编码得到mask，指导后续的语义一致性。

对于$G_1$的pixel-wise对抗损失为
$${\mathcal{L}}_{la}=\frac{1}{N_s\cdot HW}\sum _{i=1}^{N_s}\sum _{k=1}^{HW}\log (D_1(G_1(x_i^s{)}_k){)}^2+\frac{1}{N_t\cdot HW}\sum _{i=1}^{N_t}\sum _{k=1}^{HW}\log (1-D_1(G_1(x_i^t{)}_k){)}^2(7)$$
对于源域和目标域的特征mask$m_f^s$和$m_f^t$的计算是利用$D_1$的不确定度，

$G_1$提取的源域和目标域样本特征为$r_i^k=(G_1(x_i){)}_k$，$D_1$的输出为$d_i^k=D_1(r_i^k)$类似公式（1）的计算方式，每个区域的不确定度为$v(r_i^k)=H(d_i^k)$,由此得到每个样本对应的feature mask为$m_f^k=2-v(r_i^k)$，经过re-weighted后的特征为${\tilde{r}}_i^k\leftarrow r_i^k\cdot m_i^k$。

$D_2$的对抗损失为
$${\mathcal{L}}_{ma}=\mathbb{E}[\log (D_2(G_2({\tilde{f}}_i^s)))]+\mathbb{E}[1-\log (D_2(G_2({\tilde{f}}_i^t)))](8)$$

Training Loss

${\mathcal{L}}_{cls}$和${\mathcal{L}}_{reg}$为目标检测的损失，总的目标函数为
$$\underset{D_1,D_2,D_3}{\max }\underset{G_1,G_2,G_3}{\min }{\mathcal{L}}_{cls}+{\mathcal{L}}_{reg}-\lambda ({\mathcal{L}}_{la}+{\mathcal{L}}_{ma}+{\mathcal{L}}_{ga}+{\mathcal{L}}_{ins})(9)$$

Experiments

Cityscapes to Foggy-Cityscapes

PASCAL VOC to Clipart

Sim10K to Cityscapes

Ablation Study