Image-to-Image Translation with Conditional Adversarial Networks 总结

cGAN：

• Conditional GAN — 在生成模型G和判别模型D中都加入条件信息来引导模型的训练
• a general-purpose solution to image-to-image translation problems
• learn the mapping from input image to output image
• learn a loss function to train this mapping.
• without the need for parameter tweaking
• without hand-engineering our loss functions

1.Introduction

• problems：translating an input image into a corresponding output image
• Traditionally：tackled with separate, special-purpose machinery(用单独、特殊用途机制来处理)
• GAN：instead specify only a high-level goal
• cGANs ：suitable for image-to-image translation tasks, where we condition on an input image and generate a corresponding output image. （image-conditional GANs）

2. Related work

Structured losses for image modeling

• Image-to-image translation problems —> per-pixel classification or regression
• Conditional GANs instead learn a structured loss
• Structured losses penalize the joint configuration of the output
• conditional GAN loss is learned, can penalize any possible structure that differs between output and target.

Conditional GANs

• Prior ：used GANs for image-to-image mappings, but only applied the GAN unconditionally, relying
  on other terms (such as L2 regression) to force the output to be conditioned on the input
• architectural choices
  generator ： “U-Net”-based architecture
  discriminator：convolutional “PatchGAN” classifier----penalizes structure at the scale of image patches.

3.Method

GANs ： G : z —> y
conditional GANs： G : { x , z } —> y

• Objective

• test the importance of conditioning the discriminator

• it beneficial to mix the GAN objective with a more traditional loss
  explore: L1 encourages less blurring

• find generator simply learned to ignore the noise
  provide dropout noise
  applied on several layers of our generator at both training and test time.
  observe only minor stochasticity(次要特征变化)
  Designing produce highly stochastic output, capture the full entropy of the conditional distributions they model, is an important question left

• Network architectures ：convolution-BatchNorm-ReLu

• Generator with skips

• structure in the input is roughly aligned with structure in the output. （大致一致）
  previous：used an encoder-decoder network

• circumvent the bottleneck（规避瓶颈）：add skip connections, following the general shape of a “U-Net”

• Markovian discriminator (PatchGAN)

• L1 produces blurry results on image generation problems （ capture the low frequencies）

• model high-frequency structure, relying on an L1 term to force low-frequency correctness

• In order to model high-frequencies, it is sufficient to restrict our attention to the structure in local image patches.

• design a discriminator architecture – which we term a PatchGAN – that only penalizes structure at the scale of patches.

• PatchGAN can be understood as a form of texture/style loss