BeautyGAN: Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network（ACMMM18）

3 OUR APPROACH: BEAUTYGAN

non-makeup image domain $A\subset {\mathbb{R}}^{H\times W\times 3}$，makeup image domain $B\subset {\mathbb{R}}^{H\times W\times 3}$

生成器$\left(I_{src}^B,I_{ref}^A\right)=G\left(I_{src},I_{ref}\right)$

输入包括：source image $I_{src}\in A$，reference image $I_{ref}\in B$
输出包括：after-makeup image $I_{src}^B\in B$，anti-makeup image $I_{ref}^A\in A$

3.1 Full Objective

框架图如Fig. 2所示，包括1个生成器$G$和2个判别器$D_A,D_B$

首先是判别器$D_A,D_B$的目标函数
$$\begin{array}{rl}{\mathcal{L}}_{D_A}& ={\mathbb{E}}_{I_{src}}\left[\log D_A\left(I_{src}\right)\right]\\ & +{\mathbb{E}}_{I_{src},I_{ref}}\left[\log \left(1-D_A\left(I_{ref}^A\right)\right)\right](1)\end{array}$$
$$\begin{array}{rl}{\mathcal{L}}_{D_B}& ={\mathbb{E}}_{I_{ref}}\left[\log D_B\left(I_{src}\right)\right]\\ & +{\mathbb{E}}_{I_{src},I_{ref}}\left[\log \left(1-D_B\left(I_{src}^B\right)\right)\right](2)\end{array}$$
注：公式(1)中$I_{src}^B$与$I_{src},I_{ref}$有关，所以$\mathbb{E}$的下标为$I_{src},I_{ref}$

然后给出$G$的目标函数，包括adversarial loss、cycle consistency loss、perceptual loss、makeup constrain loss
$${\mathcal{L}}_G=\alpha {\mathcal{L}}_{adv}+\beta {\mathcal{L}}_{cyc}+\gamma {\mathcal{L}}_{per}+{\mathcal{L}}_{makup}(3)$$
其中${\mathcal{L}}_{adv}$包含如下2项
$${\mathcal{L}}_{adv}={\mathcal{L}}_{D_A}+{\mathcal{L}}_{D_B}(4)$$
注：对于$G$，${\mathcal{L}}_{adv}$只涉及fake的部分

$D_A$和$D_B$需要最大化公式(1)和(2)，$G$需要最小化公式(3)

3.2 Domain-Level Makeup Transfer

We exploit domain-level makeup transfer as the foundation of instance-level makeup transfer.

本文的目标是instance级别的makeup transfer，总体框架需要先采用domain级别的transfer，再精细化到instance级别

给定一幅图像$x$，$F_l(x)\in {\mathbb{R}}^{C_l\times H_l\times W_l}$表示$x$在VGG Network中第$l$层的feature map

两个feature map之间的MSE就是perceptual loss
$${\mathcal{L}}_{per}=\frac{1}{C_l\times H_l\times W_l}\sum _{i,j,k}{E}_l(5)$$
$$E_l={\left[F_l\left(I_{src}\right)-F_l\left(I_{src}^B\right)\right]}_{ijk}^2+{\left[F_l\left(I_{ref}\right)-F_l\left(I_{ref}^A\right)\right]}_{ijk}^2(6)$$

perceptual loss的作用是保持图像变换前后的内容大致不变，cycle consistency loss的作用也是保持图像变换前后的对应关系
$$\left(I_{src},I_{ref}\right)\to G\left(I_{src},I_{ref}\right)\to G\left(G\left(I_{src},I_{ref}\right)\right)\approx \left(I_{src},I_{ref}\right)(7)$$
cycle consistency loss定义如下
$${\mathcal{L}}_{cyc}={\mathbb{E}}_{I_{src},I_{ref}}\left[dist\left(I_{src}^{rec},I_{src}\right)+dist\left(I_{ref}^{rec},I_{ref}\right)\right](8)$$
其中$\left(I_{src},I_{ref}\right)=G\left(G\left(I_{src},I_{ref}\right)\right)$，距离度量$dist(\cdot )$可取$L_1$ norm、$L_2$ norm等

3.3 Instance-level Makeup Transfer

以上loss项保证了domain级别的transfer，为了增强到instance级别的transfer，需要增加约束条件来保证makeup style consistency

We observe that facial makeup could be visually recognized as color distributions.

作者认为makeup style transfer本质上是color changing

作者利用了一种color changing的方法，Histogram Matching (HM)，作用在图像上，从而引入additional histogram loss on pixel-level，能够使得$I_{src}^B$和$I_{ref}$之间有相同的makeup style

Histogram loss.

对于original image $x$和reference image $y$，采用Histogram Matching方法生成一幅图像$HM(x,y)$，使得$HM(x,y)$的颜色分布与$y$相同，但仍保持了$x$的content

然后对$x$和$HM(x,y)$求MSE loss

Q：original image $x$指的是生成图像吗？

Face parsing.

人脸上有3块区域对makeup style的贡献最大，分别是lipsticks、eye shadow、foundation，因此只对这3块区域计算Histogram loss

使用face parsing model来获取face guidance mask $M=FP(x)$，最终得到binary的$M_{lip},M_{eye},M_{face}$，对于$M_{eye}$修正为$M_{shadow}$，3个mask见Fig.2下方的图例

Makeup loss.

最终的makeup loss包含three local histogram losses acted on lips, eye shadows and face regions
$${\mathcal{L}}_{makeup}={\lambda }_l{\mathcal{L}}_{lips}+{\lambda }_s{\mathcal{L}}_{shadow}+{\lambda }_f{\mathcal{L}}_{face}(9)$$
每一项local histogram loss定义如下
$${\mathcal{L}}_{item}={\Vert I_{src}^B-HM\left(I_{src}^B\circ M_{item}^1,I_{ref}\circ M_{item}^2\right)\Vert }_2(10)$$
$$\begin{array}{rl}& M^1=FP\left(I_{src}^B\right)(11)\\ & M^2=FP\left(I_{ref}^B\right)(12)\end{array}$$
其中$\circ$表示点乘， i t e m ∈ { l i p s , s h a d o w , f a c e } item\in\left \{ lips, shadow, face \right \} item∈{lips,shadow,face}

注：从Fig.2来看，只对有妆的图像作用makeup loss