CNN 图像增强--DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks

DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks
ICCV2017
http://people.ee.ethz.ch/~ihnatova/
https://github.com/aiff22/DPED

本文使用CNN网络对手机拍摄的图像进行图像增强。

训练数据的问题怎么解决了？
这里我们使用3个手机和一个单反同时拍摄一个物体，由此建立了一个数据库
A new large-scale dataset of over 6K photos taken synchronously by a DSLR camera and 3 low-end cameras of smartphones in a wide variety of conditions.

2 DSLR Photo Enhancement Dataset (DPED)

在三周的时间里我们采集了 2万2千张图像，包括 sony手机的 4549张图像，苹果手机的5727张图像，黑莓手机和 佳能单反 个 6015张图像。这些图像拍摄于白天，各个不同地点，不同光照和天气条件下。使用的是自动模式，所有的相机都使用默认的设置。

Matching algorithm
这些图像虽然是同时拍摄的，但是因为四个相机具有不同的视角和位置，所以需要我们进行匹配

这里我们使用 SIFT keypoints 和 RANSAC 得到映射关系 homography

Training CNN on the aligned high-resolution images is infeasible
这里我们使用 100*100的 图像块

100 original images were reserved for testing

最终我们得到的训练测试数据如下：
139K, 160K and 162K training and 2.4-4.3K test patches for BlackBerry-Canon, iPhone-Canon and Sony-Canon pairs

3 Method
下面我们设计CNN网络来完成 输入一张 low-quality photo 输出 a DSLR camera 拍摄的图像

3.1. Loss function
图像增强任务主要的困难在于 输入输出图像不能密集匹配（像素对像素）: different optics and sensors cause specific local non-linear distortions and aberrations, leading to a
non-constant shift of pixels between each image pair even after precise alignment. Hence, the standard per-pixel losses, besides being doubtful as a perceptual quality
metric, are not applicable in our case.

这里我们基于以下假设来建立损失函数：
the overall perceptual image quality can be decomposed into three independent parts: i) color quality, ii) texture quality and iii) content quality.

3.1.1 Color loss
对于颜色差异，我们使用一个 Gaussian blur，计算两者的欧式距离，在CNN中对应一个额外得卷积层，它包含 a fixed Gaussian kernel followed by the mean squared error
(MSE) function
Color loss 定义如下：

The idea behind this loss is to evaluate the difference in brightness, contrast and major colors between the images while eliminating texture and content comparison

The crucial property of this loss is its invariance to small distortions.

color loss forces the enhanced image to have the same color distribution as the target one, while being tolerant to small mismatches

3.1.2 Texture loss
这里我们使用一个 generative adversarial networks(GANs) 来学习一个合适的 texture quality 度量

3.1.3 Content loss
参考文献【9,12】 we define our content loss based on the activation maps produced by the ReLU layers of the pre-trained VGG-19 network

3.1.4 Total variation loss
最后我们还加了一个 total variation (TV) loss 来 enforce spatial smoothness of the produced images

3.1.5 Total loss

总的网络结构图如下：

4 Experiments