DIC(Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark)

文章题目：

Deep Face Super-Resolution with Iterative Collaboration between Attentive
Recovery and Landmark Estimation

文章地址：https://arxiv.org/pdf/2003.13063.pdf

项目地址：https://github.com/Maclory/Deep-Iterative-Collaboration

Abstract：

现有的方法并没有充分利用先验知识，因为先验知识（landmark and component maps）总是由低分辨率或粗略的超分辨率图像来估计，导致恢复不准确。

However , the prior knowledge is not fully exploited in existing methods, since facial priors such as landmark and component maps are always estimated by low-resolution or coarsely super-resolved images, which maybe inaccurate and thus affect the recovery performance.

在本文中提出了一种深度人脸超分辨率（FSR）方法，该方法在两个递归网络之间进行迭代协作，分别关注人脸图像恢复和地标估计。

In this paper , we propose a deep face super-resolution (FSR) method with iterative collaboration between two recurrent networks which focus on facial image recovery and land-mark estimation respectively.

在每个循环步骤中，恢复分支利用landmark的先验知识来生成更高质量的图像，从而促进更准确的landmark估计。

In each recurrent step, the recovery branch utilizes the prior knowledge of landmarks to yield higher-quality images which facilitate more accurate landmark estimation in turn.

此外，还设计了一个新的注意融合模块（AFM），以加强landmark map的指导，在landmark map中，面部组件被单独生成并仔细聚合，以便更好地恢复。定量和定性实验结果表明，该方法在恢复高质量人脸图像方面明显优于现有的FSR方法。

Moreover , a new attentive fusion module is designed to strengthen the guidance of landmark maps, where facial components are generated individually and aggregated attentively for better restoration. Quantitative and qualitative experimental results show the proposed method significantly outperforms state-of-the-art FSR methods in recovering high-quality face images.

通过本文的框架，SR图像的质量逐渐提高，估计的地标（红色）也更接近地面真实值（绿色）。Through our framework, the quality of SR images becomes better progressively and the estimated landmarks (red) also get closer to the ground-truth (green).

2.Related Work:

本文中提出了一种深度迭代协作的人脸超分辨方法

In this paper, we propose a deep iterative collaboration method for face super-resolution.

首先，我们设计了一个新的框架，包括两个分支，一个用于face recovery，另一个用于landmark estimation。与以前的方法不同，我们让人脸SR和对齐过程逐步相互促进。

Firstly, we design a new framework including two branches, one for face recovery and the other for landmark estimation. Different from previous methods, we let the face SR and alignment processes facilitate each other progressively.

为了实现这一目标，我们为SR构建了一个递归体系结构，而不是非常深入的生成模型，同时设计了一个用于人脸对齐的递归沙漏网络。

To achieve this goal, we build a recurrent architecture instead of very deep generative models for SR while designing a recurrent hourglass network for face alignment.

其次，我们提出了一个新的注意融合模块来整合地标信息，而不是级联操作。

Secondly, we propose a new attentive fusion module to integrate the landmark information instead of the concatenation operation.

3Approach:

3.1.Deep Iterative Collaboration:

深度迭代协作方法的总体框架:

递归SR分支G由低分辨率特征提取器G1、递归块GR和高分辨率生成层G2组成。GR包括注意融合模块和重复SR模块。

The recurrent SR branch G consists of a low-resolution feature extractor G1, a recursive block GR and high-resolution generation layers G2. GR includes an attentive fusion module and a recurrent SR module.

递归对齐分支包括预处理块A1、递归沙漏块A1和后处理块A2

the recurrent alignment branch includes a pre-processing block A1, a recursive hourglass block AR and a post-processing block A2.

人脸SR分支过程公式化：

 表示第n步的SR图， 和  表示上一步的反馈信息，U 表示上采样操作。

对齐分支：

像素损失函数：

 

 分别是人脸SR(5)以及landmark estimation(6)的损失函数。

3.2.Attentive Fusion Module：

上图左侧部分说明了从Landmarks maps中提取attention maps的方法。图右侧显示了注意融合模块的流程图，输入特征由卷积层扩展，然后在attention maps的指导下，通过一系列组卷积层提取特定成分的特征。我们乘（”⊗”) 这些特征和attention maps，最后将加权特征相加形成输出。

我们通过一个新的结构感知注意融合模块来实现上述目标，以充分利用landmarks L的引导。我们假设每个地标热图都有K个通道，指示K个地标的位置。这些landmarks包括左眼、右眼、鼻子、嘴和下颌线在内的面部组件。

we achieve the above goals by a new structure-aware attentive fusion module so as to make full use of the guidance of landmarks L. We assume each landmark heatmap has K channels indicating the locations of K landmarks.The landmarks can be grouped intoPsubsets, belonging to facial components including left eye, right eye, nose, mouth and jawline.

再通过SoftMax变换：

得到图三左侧的注意图。

3.3.Objective Functions： 

对抗性损失：

 同时，生成器试图愚弄鉴别器并最小化：

感知损失：

 

总体目标（DICGAN的损失）： 

 其中λ表示表示对抗性损失和感知损失的权衡参数。

以PSNR为驱动时，训练得到的模型为DIC，此时的两个λ都为0。

4.Experiments