Towards Open-Set Identity Preserving Face Synthesis

[1803.11182] Towards Open-Set Identity Preserving Face Synthesis

Abstract

Previous identity preserving face synthesis processes are largely confined to synthesizing faces with known identities that are already in the training dataset. Our proposed framework does not need to annotate the attributes of faces in any way. It is trained with an asymmetric loss function to better preserve the identity and stabilize the training process. It can also effectively leverage large amounts of unlabeled training face images to further improve the fidelity of the synthesized faces for subjects that are not presented in the labeled training face dataset.

1. 之前的保留身份的人脸合成过程都限制于数据集中已知身份的人脸合成，而本文可训练数据集外的人脸。

2. 本文提出的架构不需要对人脸属性做任何标注。

3. 本文采用非对称函数来更好的保留身份信息和使训练过程稳定。

4. 可高效率利用大量的无标签脸部图片来提高合成的质量。

1. Introduction

（1）与其他网络对比

Many previous works have attempted to synthesize face images of a specific person. For example, TP-GAN [14] and FF-GAN [36] attempt to synthesize the frontal view of a face from a single face image. DR-GAN [33] can change the pose of an input face image. However, these methods can only manipulate limited types of attributes, such as poses. These methods also require full annotation of attributes for training the models. More recent work, such as CVAE- GAN [4], can produce a variety of attribute changes. Nevertheless, it is not able to synthesize a face with an identity outside the training dataset.

1. 许多之前的工作尝试合成特定的人脸。比如TP-GAN, FF-GAN（通过单一脸部图像合成正脸图像）, DR-GAN（改变输入脸部图像的表情）。

   但这些方法都仅能修改某种限定属性，而且要求输入图片属性的所有标注来训练模型。

2. CVAE-GAN可以修改很多属性，但不能合成来自训练集以外的人脸。

（2）本文原理

   To synthesize a face with an identity outside the training dataset, we require one input image of that subject to produce an identity vector, and any other input face image to extract an attribute vector capturing, e.g., pose, emotion, illumination, and even background. We then combine the identity vector and the attribute vector to synthesize a new face of the subject with the extracted attribute.
   To this end, we propose a framework based on Generative Adversarial Networks to disentangle identity and attributes given a face image, and recombine different identities and attributes for identity preserving face synthesis.

我们需要一个主体的输入图像来提供一个身份向量，和其他输入的脸部图像来提取一个属性向量捕获，比如姿态、感情等。然后我们将这些身份向量和提取到的属性向量合成来合成主体的新脸部。

末端的GAN用来分离给定脸部图片的身份和属性，并将不同的身份和属性重组来做保留身份的脸部合成。

框架由五部分组成：1）身份编码网络I ；2）属性编码网络A；3）生成网络G，通过组合身份和属性来合成人脸；4）分类网络C，保留生成的人脸的身份信息；5）判别网络D，区分真实的和生成的实例。这五个部分的训练方式是端对端。 （3）损失函数

We use two loss functions: 1) a reconstruction loss of the attribute image, and 2) a KL divergence loss defined on the attribute vector. These functions enforce that network A extracts the attribute information.We take full advantage of recent advancements in face recognition, and use the softmax loss on top of network I to encode the identity into an attribute independent vector representation. Therefore, in order to reconstruct the input, network A is forced to extract the attribute information. Meanwhile, we add a KL divergence loss to regularize the attribute vector, such that it dose not contain identity information.
Inspired by the CVAE-GAN [4], we adopt a new asymmetric loss function. More specifically, we adopt a cross-entropy loss when training the discriminative network D, and the classification network C, and use a pairwise feature matching loss when updating the generative network G. This does a better job of preserving the identity while stabilizing the training process.

Ⅰ. 使用了两个损失函数：

（1）定义在属性图片上的重构损失。在I网络的顶端使用softmax损失函数来将身份编码为一个独立于属性的向量表示。

（2）定义在属性向量上的KL散度损失。在提取属性信息的A网络上加入一个KL散度损失来正则化属性向量，使其不包含身份信息。

Ⅱ. 采用了一种新的不对称的损失函数。

（1）在训练判别器D和分类器C的时候采用交叉熵损失；

（2）在更新生成器G的时候采用成堆的特征匹配损失。

本文的网络可以高效利用大量的无标签脸部训练图片来提高合成数据集中没有的脸的保真度。这些无标签数据可以增大类内和类间人脸分布差异以提高合成人脸的多样性，所以生成的人脸在姿态和表情上有更大的变化。

2. Related work

In contrast, this paper proposes an Identity PreservingGenerative Adversarial Network framework, which does not require any attribute annotations. This framework disentangles the identity and attributes representations, and then uses different recombinations of representations for identity preserving face synthesis. This disentaglement allows us to synthesize faces with identities outside what is presented in the training datasets. This addresses a serious limitation of a previous deep generative model-based identity preserving face synthesis method [4]. It simply can not generate faces of identities outside the training dataset.

对比之下，本文提出了一种保留身份的GAN，它并不要求任何属性注释。本文的结构将身份和属性分离，然后使用不同表示形式的组合来做保留身份的人脸合成。这种分离使我们可以合成训练集中没有的人脸，而其他的方法不能。

3. 保留身份信息的GAN

实线：输入输出 虚线：损失函数
$I$从图片$x^s$中提取身份向量$f_I(x^s)$。从属性图片中提取属性向量$f_A(x^a)$。G通过组合身份向量和属性向量$[f_I(x^s{)}^T,f_A(x^a{)}^T{]}^T$来生成一个新的脸部图片$x^{\prime }$。C通过后验概率$P(c|x^s)$来保留身份信息，其中c是身份信息x^s的主体（类别/标签）。判别网络D用来区分真实图片和生成图片。

3.1 身份信息和属性信息的分离

In our training data, we only have the annotation of the identity of each face, without any annotation of the attribute information. Extracting the identity vector is relatively straightforward. Here, we take full advantage of recent improvement in face recognition.

训练数据有身份信息的标注，没有属性信息的标注。

3.1.1 提取身份向量

给定脸部图片的身份信息 { x i s , c i } （ x i s \{x_i^s,c_i\}（x_i^s {