Landmark Assisted CycleGAN for Cartoon Face Generation

3. Our Method

3.1. Review of CycleGAN

给定来自两个domain的unpaired training samples $x\in X,y\in Y$，对于其从$X$到$Y$的mapping $G_{X\to Y}$，及其判别器$D_Y$，adversarial loss定义如下
$$\begin{array}{rl}{\mathcal{L}}_{GAN}& \left(G_{X\to Y},D_Y\right)={\mathbb{E}}_y\left[\log D_y(y)\right]\\ & +{\mathbb{E}}_x\left[\log \left(1-D_Y\left(G_{X\to Y}(x)\right)\right)\right](1)\end{array}$$

CycleGAN学习正向和反向的mapping，the cycle consistency
loss如下
$$\begin{array}{rl}{\mathcal{L}}_{cyc}=& {\Vert G_{Y\to X}\left(G_{X\to Y}(x)\right)-x\Vert }_1+\\ & {\Vert G_{X\to Y}\left(G_{Y\to X}(y)\right)-y\Vert }_1(2)\end{array}$$

CycleGAN的total objective function定义如下
$$\begin{array}{rl}\mathcal{L}(G_{X\to Y},& G_{Y\to X},D_X,D_Y)=\\ & {\mathcal{L}}_{GAN}\left(G_{X\to Y},D_Y\right)+\\ & {\mathcal{L}}_{GAN}\left(G_{X\to Y},D_Y\right)+{\mathcal{L}}_{cyc}(3)\end{array}$$

本文定义$X$为real face domain，$Y$为cartoon face domain

3.2. Cartoon Face Landmark Assisted CycleGAN

3.2.1 Landmark Consistency Loss
$$\begin{array}{rl}{\mathcal{L}}_c(& G_{(X,L)\to Y})=\\ & {\Vert R_Y\left(G_{(X,L)\to Y}\left(x,l\right)\right)-l\Vert }_2(4)\end{array}$$
其中 $l\in L$是input landmark heatmap，$R$是一个预训练的U-Net，用于预测landmark heatmap，$R_Y$表示domain $Y$中的landmark regressor

公式(4)的含义为，对于real face image $x$及其landmark $l$，送入生成器$G_{(X,L)\to Y}$生成图像，对于生成的图像使用$R_Y$预测landmark，应该尽可能地与$l$接近

3.2.2 Landmark Matched Global Discriminator
如Figure 2所示，对于translation $X\to Y$，unconditional global discriminator $D_Y$ produces more realistic cartoon faces，conditional global discriminator $D_Y^{g_c}$ aims to generate landmark-matched cartoon faces with landmark heat map $l\in L$ as part of input
$$\begin{array}{rl}{\mathcal{L}}_{GAN}(& G_{(X,L)\to Y},D_Y^{g_c})={\mathbb{E}}_y\left[\log D_Y\left(y,l\right)\right]\\ & +{\mathbb{E}}_x\left[\log \left(1-D_Y\left(G_{(X,L)\to Y}\left(x,l\right),l\right)\right)\right](5)\end{array}$$

3.2.3 Landmark Guided Local Discriminator
在眼睛、鼻子、嘴巴的区域引入3个local discriminators，其adversarial loss定义如下
$$\begin{array}{rl}& {\mathcal{L}}_{GA{N}_{local}^{X\to Y}}=\sum _{i=1}^3{\lambda }_{l_i}\cdot {\mathcal{L}}_{GA{N}_{patch}}\left(G_{(X,L)\to Y},D_Y^{l_i}\right)\\ & =\sum _{i=1}^3{\lambda }_{l_i}\{{\mathbb{E}}_y\left[\log D_Y^{l_i}\left(y_p\right)\right]\\ & +{\mathbb{E}}_x\left[\log \left(1-D_Y^{l_i}\left({\left[G_{(X,L)\to Y}(x)\right]}_p\right)\right)\right]\}(6)\end{array}$$
其中$y_p$与${\left[G_{(X,L)\to Y}(x)\right]}_p$分别表示real cartoon image与generated cartoon image的local patch

3.3. Network Training

3.3.1 Two Stage Training

Stage I 首先在framework中去掉local discriminator训练100K iterations，得到coarse results

Stage II 使用pre-trained landmark prediction network对coarse images预测landmark，利用landmark提取local patch，送入local discriminator得到更精确的生成结果