LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeup（ICCV19）

3. LADN

3.1. Problem Formulation

定义domain，$X\subset {\mathbb{R}}^{H\times W\times 3}$为before-makeup faces，$Y\subset {\mathbb{R}}^{H\times W\times 3}$为after-makeup faces

数据集包括 { x i } i = 1 , ⋯ M , x i ∈ X \left \{ x_i \right \}_{i=1,\cdots M}, x_i\in X {xi​}i=1,⋯M​,xi​∈X以及 { y i } i = 1 , ⋯ N , y i ∈ Y \left \{ y_i \right \}_{i=1,\cdots N}, y_i\in Y {yi​}i=1,⋯N​,yi​∈Y

goal：学习化妆的mapping function ${\Phi }_Y:x_i,y_j\to {\tilde{y}}_i$，以及卸妆的mapping function ${\Phi }_X:y_j\to {\tilde{x}}_j$
值得注意的是，makeup的过程是需要reference image作为condition，而makeup removal的过程不需要condition

3.2. Network Architecture

核心idea：separate the makeup style latent variable from non-makup features (identity, facial structure, head pose, etc.) and generate new images through recombination of these latent variables.
因此本文借鉴了一个disentanglement framework DRIT

定义：attribute space $A$ that captures the makeup style latent、content space $S$ which includes the non-makeup features

网络：2个domain的content encoders { E X c , E Y c } \left \{ E_X^c, E_Y^c \right \} {EXc​,EYc​}，style encoders { E X a , E Y a } \left \{ E_X^a, E_Y^a \right \} {EXa​,EYa​}（上标为$a$，其实就是attribute encoders），generators { G X , G Y } \left \{ G_X, G_Y \right \} {GX​,GY​}

使用Encoder网络分别对$x_i$和$y_j$提取attribute and content features
$$\begin{gathered} E_X^a(x_i)=A_i{E}_Y^a(y_j)=A_j \\ {E}_X^c(x_i)=C_i{E}_X^c(y_j)=C_j \end{gathered}$$
然后送入generators生成de-makeup result ${\tilde{x}}_j$和makeup transfer result ${\tilde{y}}_i$
$$G_X\left(A_i,C_j\right)={\tilde{x}}_j{G}_Y\left(A_j,C_i\right)={\tilde{y}}_i(1)$$

The encoders and decoders are designed with a U-Net structure, The latent variables $A$, $C$ are concatenated at the bottleneck and skip connections are used between the content encoder and generator. This structure can help retain more identity details from the source in the generated image.

对于2个domain，设置2个判别器 { D X , D Y } \left \{ D_X, D_Y \right \} {DX​,DY​}，从而有adversarial loss $L_{domain}^{adv}=L_X^{adv}+L_Y^{adv}$
$$\begin{array}{rl}& L_X^{adv}={\mathbb{E}}_{x\sim P_X}\left[\log D_X(x)\right]+{\mathbb{E}}_{\tilde{x}\sim G_X}\left[\log \left(1-D_X\left(\tilde{x}\right)\right)\right]\\ & L_Y^{adv}={\mathbb{E}}_{y\sim P_Y}\left[\log D_Y(y)\right]+{\mathbb{E}}_{\tilde{y}\sim G_Y}\left[\log \left(1-D_Y\left(\tilde{y}\right)\right)\right](2)\end{array}$$

3.3. Local Style Discriminator

除了2个global的判别器$D_X,D_Y$，额外引入local discriminator

对于每一个unpaired的样本$\left(x_i,y_j\right)$，人工生成一个synthetic ground truth $W\left(x_i,y_j\right)$，方法是warping and blending $y_j$ onto $x_i$ according
to their facial landmarks

当然warping result是有artifacts的，需要使用网络来fix

注：这个生成的代价是否太大了，假设$X$有1k幅图像，$Y$有1k幅图像，那么两两交叉，需要生成100万幅图像

好处是：Although the synthetic results cannot serve as the real ground truth of the final results, they can provide guidance to the makeup transfer network on what the generated results should look like.

如Fig.3所示，对于512x512尺寸的reference image $y_j$，warping result $W\left(x_i,y_j\right)$，generated result ${\tilde{y}}_i$上，设置$K=12$个box框出不同的patch，每个box的尺寸为102x102，这些box的位置是与facial landmarks绑定的

以红色的box为例，(a)~(c)展示了patch放大后的样子

$K$个patch对应了$K$个local discriminators ${\left\{D_k^{local}\right\}}_{k=1,\cdots ,K}$

Note that the local discriminators are overlapping.（因为box是有公共区域的，所以说local discriminator是overlapping）

将reference image上的patch $p_k^Y$，warping result上的patch $p_k^w$，generated result上的patch ${\tilde{p}}_k^Y$，组合成positive pair $\left(p_k^Y,p_k^w\right)$，negative pair $\left(p_k^Y,{\tilde{p}}_k^Y\right)$，送入local discriminator $D_k^{local}$

定义local adversarial loss $L^{local}={\sum }_k{L}_k^{local}$
$$\begin{array}{rl}{L}_k^{local}& ={\mathbb{E}}_{x_i\sim P_X,y_j\sim P_Y}\left[\log D_k^{local}\left(p_k^Y,p_k^W\right)\right]\\ & +{\mathbb{E}}_{x_i\sim P_X,y_j\sim P_Y}\left[\log \left(1-D_k^{local}\left(p_k^Y,{\tilde{p}}_k^Y\right)\right)\right](3)\end{array}$$

local discriminator与生成器网络之间的min-max game定义如下
$$\underset{D_k^{local}}{\max }\underset{E_X^c,E_Y^a,G_Y}{\min }{L}^{local}(4)$$

总的来说，人工生成warping result以及引入local discriminator的目的为了增加guidance，以帮助生成器更好地capture makeup details from the makeup reference

3.4. Asymmetric Losses

对于extreme makeup style（类似彩绘一样的妆容），仍然是具有挑战性的，原因在于：

1. 对于化妆过程，extreme makeup style包含了high-frequency components，需要网络将这些高频成分与其它脸部高频成分区（如睫毛）分开来
2. 对于卸妆过程，需要网络重构出人脸原有的正常肤色（hallucinate the
   facial skin color without makeup），hallucinate意味着需要一定的想象力

因此，对于makeup transfer branch引入high-order loss $L^{ho}$来提升对high-frequency details的迁移性能；对于de-makeup branch引入smooth loss $L^{smooth}$来提升重构人脸原有肤色的能力

High-Order Loss
将Laplacian filters作用到$p_k^W,{\tilde{p}}_k^Y$上，定义high-order loss如下
$$L^{ho}=\sum _k{h}_k{\Vert f\left(p_k^W\right)-f\left({\tilde{p}}_k^Y\right)\Vert }_1(5)$$
其中$h_k$是每一个local patch分配的权重，$f$是Laplacian filter
对于所有的local patch都设置相等的$h_k$，除了眼部区域的权重稍微大一点

Smooth Loss
与makeup transfer相反，我们不希望de-makeup result ${\tilde{x}}_j$中表现出high-frequency details，换言之，我们希望${\tilde{x}}_j$的local patch是平滑的，因此为所有patch定义smooth loss如下
$$L^{smooth}=\sum _k{s}_k{\Vert f\left({\tilde{p}}_k^X\right)\Vert }_1(6)$$
其中${\tilde{p}}_k^X$是${\tilde{x}}_j$中的local patch，$s_k$是每一个local patch分配的权重，$f$是Laplacian filter
对于眼部区域的权重设置得小一些，因为我们不希望丢失眼部的高频成分；对于脸颊和鼻子区域，设置比较大的权重

总的来说，把high-order loss和smooth loss统称为asymmetric losses，表明二者的目的是相反的，high-order loss鼓励网络迁移高频成分的能力，smooth loss追求平滑的生成结果，希望抑制高频成分

3.5. Other Loss Functions

Reconstruction Loss
借鉴CycleGAN的cycle consistency，对于Fig.2，之前的做法是将分离出来的attribute和content做交叉，现在的做法是不做交叉，将$\left(A_i,C_i\right)$送入$G_X$生成${\tilde{x}}_i^{self}$，将$\left(A_j,C_j\right)$送入$G_Y$生成${\tilde{y}}_j^{self}$，于是${\tilde{x}}_i^{self}\approx x_i,{\tilde{y}}_j^{self}\approx y_j$

另一方面，对于generated results ${\tilde{x}}_j,{\tilde{y}}_i$，再次提取attribute和content做交叉，生成${\tilde{x}}_i^{cross},{\tilde{y}}_j^{cross}$，它们应该与$x_i,y_j$相等，称为cross-cycle reconstruction loss
Q：这个loss是否来自DRIT？

所有的reconstruction loss定义如下
$$\begin{array}{rl}{L}_{reocn}=& {\Vert x_i-{\tilde{x}}_i^{self}\Vert }_1+8{\Vert x_i-{\tilde{x}}_i^{cross}\Vert }_1+\\ & {\Vert y_j-{\tilde{y}}_j^{self}\Vert }_1+8{\Vert y_j-{\tilde{y}}_j^{cross}\Vert }_1(7)\end{array}$$

KL Loss
令网络提取的attribute信息 { A i , A j } \left \{ A_i, A_j \right \} {Ai​,Aj​}近似于高斯分布，于是引入KL loss $L^{KL}=L_i^{KL}+L_j^{KL}$，其中
$$\begin{array}{rl}& L_i^{KL}=\mathbb{E}\left[\left(D_{KL}\left(A_i\parallel N(0,1)\right)\right)\right]\\ & L_j^{KL}=\mathbb{E}\left[\left(D_{KL}\left(A_j\parallel N(0,1)\right)\right)\right]\\ & D_{KL}(p\parallel q)=\int p(x)\log \left(\frac{p(x)}{q(x)}\right)dx(8)\end{array}$$
Q：为什么attribute需要近似于高斯分布，而content就不需要呢？

Total Loss
$$\begin{array}{rl}{L}^{total}=& {\lambda }_{local}{L}^{local}+{\lambda }_{domain}^{adv}{L}_{domain}^{adv}+{\lambda }_{recon}{L}^{recon}+\\ & {\lambda }_{KL}{L}^{KL}+{\lambda }_{ho}{L}^{ho}+{\lambda }_{smooth}{L}^{smooth}(9)\end{array}$$