论文在Image Style Transfer Using Convolutional Neural Networks。
风格迁移一个比较火的应用是Prisma,关于其的介绍可参考AI修图艺术:Prisma背后的奇妙算法 | 深度,不过这篇论文的一个硬伤就是,Prisma里面的原理不是用这篇文章里的,这篇文章里有写:
其中alpha和belta分别代表风格和内容的权重比例,若alpha/belta高则生成图会更凸显出内容而风格化会少一些,低则生成图会风格化强烈但是内容被稀释,这也是 Prisma提供用户调节的原理所在。
然而,调整这个权重比例后需要重新训练,而Prisma的训练都是在云端进行的,网络延迟是不允许很畅快的调整的,有兴趣的可以把Prisma下了玩一玩,还是很有趣的一个应用。
Prisma 背后的算法是怎样炼成的?它有什么其他应用价值?里面有比较专业的分析,还在评论里发现了论文Demystifying Neural Style Transfer的作者Naiyan Wang,都是大神啊,这篇论文也在读的计划之中(最近发现风格迁移的作用不大,主要是风格迁移的效果无法量化,因此读的计划无限搁置吧)。
知乎这篇文章里有个回答对风格迁移讲的很明白,这里摘抄如下:
卷积神经网络(CNN)可以学习到图像的各种特征:
1.底层(靠近输入层)可以获取诸如局部模式、边缘、描边等低层次信息;
2.高层(靠近输出层)可以获取物体部件和特征等高层次信息;
风格迁移做的就是利用算法使输出图像的底层表征与指定“风格”的图片相同,同时保持原图片的高层次表征。使得转换后的图像看起来像原图,但是拥有不同的风格。
0. Abstract
风格迁移的一个关键是讲内容和风格分离出来,这个是贯穿全文,请牢记于心:
Arguably, a major limiting factor for previous approaches has been the lack of image representations that explicitly represent semantic information and, thus, allow to separate image content from style.
1. Introduction
风格迁移要将指定“风格”的图片的纹理特征迁移到输出图像中,同时保持输出图像的语义内容:
Transferring the style from one image onto another can be considered a problem of texture transfer. In texture transfer the goal is to synthesise a texture from a source image while constraining the texture synthesis in order to preserve the semantic content of a target image.
之前的算法只会利用底层特征,而理想情况下的风格迁移还会保留目标图像的语义信息。
论文讲的是利用CNN来同时获得底层特征和高级特征。
2. Deep image representations
论文利用的是VGG-19网络,对权值进行了正则化:
We normalized the network by scaling the weights such that the mean activation of each convolutional filter over images and positions is equal to one.
不是很明白论文说该正则化不会影响网络的输出:
Such re-scaling can be done for the VGG network without changing its output, because it contains only rectifying linear activation functions and no normalization or pooling over feature maps.
论文将Maxpool换成了Average pool.
2.1. Content representation
首先弄懂几个变量代表的意思吧:
A layer withdistinct filters has
feature maps each of size
, where
is the height times the width of the feature map.
这里先贴一张论文的图吧:
![3f82160134d454560464f6a4f3c0d81e.png](https://i-blog.csdnimg.cn/blog_migrate/ffe9087ed16ea0e80cff49f3ce06eb79.jpeg)
开始对Content Reconstructions不太理解,姑且不论中间经过了pool层,经过卷积后图像就会变化很多的,但是重建后的图像和原图很像,当时我一直不太明白重建的意思,后来发现论文写的还是很明白的:
To visualise the image information that is encoded at different layers of the hierarchy one can perform gradient descent on a white noise image to find another image that matches the feature responses of the original image.
什么意思呢,首先想想反步法,反步法的基本推导公式依靠的是
之后定义了损失函数
有了损失函数,之后就可以使用反步法了:
Thus we can change the initially random imageuntil it generates the same response in a certain layer of the Convolutional Neural Network as the original image
.
之后对低层特征和高级特征的分析也是相当精彩的,低层特征能够重建准确的像素,高级特征只能得到high-level content,所以会使用高级特征的层作为content representation。
2.2. Style representation
论文讲的Gram matrix不太了解原理是什么:
It consists of the correlations between the different filter responses, where the expectation is taken over the spatial extent of the feature maps.
其作用是:
By including the feature correlations of multiple layers, we obtain a stationary, multi-scale representation of the input image, which captures its texture information but not the global arrangement.
然后就可以得到每一层的loss:
The separation of image content from style is not necessarily a well defined problem. This is mostly because it is not clear what exactly defines the style of an image. It might be the brush strokes in a painting, the colour map, certain dominant forms and shapes, but also the composition of a scene and the choice of the subject of the image - and probably it is a mixture of all of them and many more.
最后的偏导使用的技巧就是
大于0的推导是:
而
2.3. Style transfer
生成风格迁移图的思想就是让新的图与内容图和风格图同时都接近:
To transfer the style of an artworkonto a photograph
we synthesise a new image that simultaneously matches the content representation of
and the style representation of
. Thus we jointly minimise the distance of the feature representations of a white noise image from the content representation of the photograph in one layer and the style representation of the painting defined on a number of layers of the Convolutional Neural Network.
![41239364bd5844384b27e563293b2722.png](https://i-blog.csdnimg.cn/blog_migrate/dff4d152b6982c4aa6df3d07cda7a609.jpeg)
总的loss函数是:
L-BFGS可以参考数值优化:理解L-BFGS算法-码农场,其所用数学知识较复杂,待闲下来再详细看看。
3. Results
主要进行了一些实验,自己看看就好。
3.1. Trade-off between content and style matching
分析了
3.2. Effect of different layers of the Convolutional Neural Network
比较了卷积层对结果的影响。
3.3. Initialisation of gradient descent
比较了初始化图像对结果的影响。
3.4. Photorealistic style transfer
生成的图片很真实!
4. Discussion
训练速度比较慢,在K40上要训练一个小时。
对噪声很敏感。
一个遗憾就是风格迁移的效果很难量化,这是与科学的基本精神背道而驰的:
We are fully aware though that this evaluation criterion is neither mathematically precise nor universally agreed upon.
[已完结]