项目总结
1.简述思想
本项目主要是利用预训练好的GAN生成器(StyleGAN)作为先验实现低质量人脸图片的修复
论文中提到:
(1) We leverage rich and diverse generative facial priors for blind face restoration. Those priors contain sufficient facial tex-tures and color information, allowing us to jointly perform face restoration and color enhancement.
即先验信息包含面部纹理与颜色信息,使得面部恢复与颜色增强得以实现。
(2) We propose the GFP-GAN framework with delicate designs of architec-tures and losses to incorporate generative facial prior. Our GFP-GAN with CS-SFT layers achieves a good balance of fidelity and texture faithfulness in a single forward pass.
GFPGAN框架使用的损失十分丰富
2.网络结构分析
(1)判别器部分
判别器既拥有对整个人脸的判别损失结算,也包含了对部分组件即左右眼和嘴部进行局部的判别损失计算。
(2)生成器部分
生成器部分包含两部分:
- 蓝色部分,退化移除模块(Degradation Removal):U-net结构
- 绿色部分,预训练好的Style-GAN
使用MLP(多层感知机)的Flatent 与Fspatial (Unet的decoder部分获得的多个特征图)来实现两个模块之间的连接
而灰色部分是计算Restoration loss来获得较为干净的特征,代码分析过程中也分析到了具体的实现在
gfpganv1_clean_arch.py中每一层都是如下操作,都会计算一个Restoration loss
现在我们可以结合之前的代码分析看一下过程:
- 图片经过退化移除模块(去除模糊噪声等,提取较为干净的特征)得到F_latent,
- F_latent经过多层感知机获得16个维度为512的 latent code
# style code style_code = self.final_linear(feat.view(feat.size(0), -1)) if self.different_w: style_code = style_code.view(style_code.size(0), -1, self.num_style_feat) # style codes -> latents with Style MLP layer if not input_is_latent: styles = [self.style_mlp(s) for s in styles]
- 这些latent codes会被送入Style-GAN生成F_GAN(intermediate convolutional features)
-
Style-GAN生成的Channel-Split SFT需要做空间变换(平移缩放),如图黄色部分,具体代码:
# decode for i in range(self.log_size - 2): # add unet skip feat = feat + unet_skips[i] # ResUpLayer feat = self.conv_body_up[i](feat) #通过下面的两层卷积,做缩放和位移 # generate scale and shift for SFT layer scale = self.condition_scale[i](feat) conditions.append(scale.clone()) shift = self.condition_shift[i](feat) conditions.append(shift.clone()) # generate rgb images if return_rgb: out_rgbs.append(self.toRGB[i](feat)) #两层卷积神经网络的具体实现 self.condition_scale.append( nn.Sequential( nn.Conv2d(out_channels, out_channels, 3, 1, 1), nn.LeakyReLU(0.2, True), nn.Conv2d(out_channels, sft_out_channels, 3, 1, 1))) self.condition_shift.append( nn.Sequential( nn.Conv2d(out_channels, out_channels, 3, 1, 1), nn.LeakyReLU(0.2, True), nn.Conv2d(out_channels, sft_out_channels, 3, 1, 1)))
具体的损失函数在代码分析过程中已有,不再赘述