本文目录
- Deep Learning on Image Denosing: An Overview
- Abstract
- 1 Introduction
- 2 Fundamental frameworks of deep learning methods for image denoising
- 3 Deep learning techniques in image denoising
- 4 Experimental results
- 5 Discussion
- 6 Conclusion
Deep Learning on Image Denosing: An Overview
Abstract
基于深度学习的判别学习对于解决高斯噪声的问题十分游泳,并且对于真实的噪声也有一定的效果
本文分为几部分:白噪声图像;真实噪声图像;盲去噪;包含噪声、模糊和低分辩率的混合图像。分析了不同深度学习方法的动机和准则,与最新的方法作对比,最后指出了潜在的挑战和未来研究方向
Q:白噪声?盲去噪?
1 Introduction
相机捕捉的是已经退化的图像,受到光和噪声的干扰,噪声在传输和潜在压缩中产生
50年前非线性和非自适应滤波器被用于图像去噪,之后基于稀疏的方法被应用,为了恢复潜在图像的细节,先验知识(总变分正则)可以使得噪声图像变得光滑,虽然很多方法效果不错,但是都有各自的缺点,例如有的需要对测试集进行优化
深度学习的技术发展开来,并逐渐被改进,例如损失函数MSE到entropy损失,对偶拉格朗日乘子,贪婪算法等被用于神经网络,之后构建一个全新的神经网络结构对于去噪很有意义,不论是增强他的深度或者改变他的激活函数
卷积神经网络被提出,但是面临很多的问题,例如梯度消失,激活函数的计算损失较大,硬件设备的计算速度较慢。2012年,Alex网络被提出,在2015年,深度神经网络被应用于图像去噪,DnCNN解决多重low-level任务,包含conv、batchnorm和Relu。Resnet被提出解决去噪、超分辨和JPEG
去遮挡。为了权衡去噪的表现和速度,CNLNet被提出来有效的取出彩图噪声
FFDNet提升了去噪速度并且可以处理盲噪声。GAN的提出可以处理非配对的噪声图像
2 Fundamental frameworks of deep learning methods for image denoising
2.1 Machine learning methods for image denoising
监督学习:
y
=
x
+
μ
y = x + \mu
y=x+μ
无监督学习:给定训练样本寻找pattern而不是标签匹配
半监督学习:将模型应用于给定的数据分布来学习,然后用来给无标签样本标签
2.2 Neural networks for image denoising
神经网络包括神经元,输入 X X X,激活函数 f f f,权重 W = [ W 0 , W 1 . . . W n − 1 ] W=[W^{0}, W^{1}...W^{n-1}] W=[W0,W1...Wn−1]和偏置 b = [ b 0 , b 1 . . . b n − 1 ] b=[b^{0}, b^{1}...b^{n-1}] b=[b0,b1...bn−1]
多层神经网络被称为MLP,下图中包含两层,隐藏层和输出层
process
f
(
X
;
W
;
b
)
=
f
(
W
n
f
(
W
n
−
1
.
.
.
f
(
W
0
X
+
b
0
)
.
.
.
b
n
−
1
)
+
b
n
)
f(X;W;b) = f(W^{n}f(W^{n-1}...f(W^{0}X+b^{0})...b^{n-1})+b^{n})
f(X;W;b)=f(Wnf(Wn−1...f(W0X+b0)...bn−1)+bn)
2.3 CNNs for image denoising
LeNet—AlexNet—VGGNet—GoogLeNet
VGG
GoogLeNet
Drawbacks:网络很深可能会产生梯度消失或者梯度爆炸;过拟合——>解决ResNet
ResNet
GAN
Generator and Discriminator
2.4 Hardware and software used in deep learning
Hardware : Nvidia AMD
Software : Caffe Theano Matconvnet TensorFlow Keras PyTorch
3 Deep learning techniques in image denoising
3.1 Deep learning techniques for additive white noisy-image denoising
白噪声图片(AWNIs)
Gaussian,Poisson,Salt,Pepper,Multiplicative noisy images
3.1.1 CNN/NN for AWNI denoising
2017 DnCNN
多尺度底层视觉工作:图像去噪,超分辨,去块滤波
残差学习——深层神经网络——递归和自适应特征提取——改进CNNs,深度神经网络,残差学习,多尺度知识(计算复杂度较大,内存损耗多)——增强感受野——通过多尺度交叉路径的残差连接压制噪声
改变网络结构框架
-
从CNN的多个输入中融合特征:一个样本的不同部分作为不同网络的多个输入;一个样本作为输入的不同视角;不同通道作为CNN的输入
-
改变损失函数
-
增强CNN的深度或宽度:通过增强网络的深度和宽度来扩大感受野的大小
-
增加CNN的一些辅助插件:例如激活函数,空洞卷积(dilated convolution),全连接层和池化操作来增强CNN的表达能力
-
引入跳跃连接或级联操作:跳跃连接能增强级联操作
3.1.2 CNN/NN and common feature extraction methods for AWNI denoising
Weak edge-information noisy images
进行域与域之间的转变
non-linear noisy images
CNN提取特征;利用核函数的方法将获取的非线性特征映射到线性;利用残差学习重建潜在清晰图像
high dimensional noisy images
CNN与降维方法相结合。PCA:卷积操作提取特征,PCA降维,卷积处理降维的特征重建图像
non-salient noisy images
信号处理可以让CNN提取非显著特征,例如,跳跃连接
tasks involving high computational costs
3.1.3 Combination of optimization method and CNN/NN for AWNI denoising
discriminative learning;prior knowledge via regular term of loss function
denosing speed
GAN估计噪声,处理其他任务,图像恢复和超分辨率
基于CNN经验的贪婪算法和迁移学习能加速普遍的算法
denoising performance
基于CNN的优化方法使得噪声图片更加光滑,CNN的全变分去噪减少了噪声像素的有效性。GAN最近邻算法有效地噪声
3.2 Deep learning techniques for real noisy image denoising
single end-to-end CNN
改变网络结构。多尺度知识对图像去噪很有效。
提取细节特征:recurrent connections
处理未知真实噪声:residual structure was utilized to facilitate low-frequency features
提取通道更多潜在特征:attention
the combination of prior knowledge and CNN
3.3 Deep learning techniques for blind denoising
3.4 Deep learning techniques for hybrid noisy image denoising
混合噪声图像:噪声、模糊、低分辨率等
突发技术burst techniques:suffered from effects of noise and camera shake
4 Experimental results
4.1 Datasets
4.1.1 Training datasets
灰度噪声gray-noisy
训练高斯去噪和盲去噪:
- BSD400 dataset:
png
:400;size
:180*180 - Waterloo Exploration Database:
png
:4744 nature images
颜色噪声color-noisy
- BSD432
- Waterloo Exploration Database
- polyU-Real-World-Noisy-Images datasets:
real noisy images
:100;size
:2784*1856
4.1.2 Test datasets
灰度噪声gray-noisy
测试高斯噪声和盲噪声:
- Set12:contain 12 scenes
- BSD68:contain 68 nature images
颜色噪声color-noisy
- CBSD68
- Kodak24:24 color noisy images
- McMaster:18 color noisy images
- cc:15 real noisy images of different ISO(1600, 3200, 6400)
- DND:50 real noisy images(clean images were captured by low-ISO images)
- NC12:12 noisy images(didn’t have ground-truth clean images)
- SIDD:320 images pairs of noisy and ground-truth images(real noisy images from smart phones)
- Nam:11 scenes(saved in JPGE format)
4.2 Experimental resultes
4.2.1 Deep learning techniques for additive white noisy-image denoising
Table10:灰度图附加白噪声
Table11:FSIM
放大图片的局部区域,观察去噪效果
4.2.2 Deep learning techniques for real-noisy image denoising
Table15 16:DRDN has the best performance
Table 17:compressed noisy images. AGAN obtains excellent performance.
Table18:real noisy images of different IOS value, SDNet and BRDNet achieved the best and the second-best denoising performance.
4.2.3 Deep learning techniques for blind denoising
blind denoising: noise which is complex in the real world and not subject to rules
FFDNet and ADNet are superior to other methods in blind denoising
4.2.4 Deep learning techniques for hybrid-noisy-image denoising
deep learning techniques based multi-degradation idea:WarpNet DnCNN and MemNet
5 Discussion
Improving denoising performance kills
- Enlarge the receptive field can capture more context information: increasing the depth and width of the networks. However, this results in higher computational costs and more memory consumption——>solve: dilated convolution——>higher performance and efficiency, edge information
- extra information(prior knowledge) combined with CNN: loss function
- combine local and global information: enhance the memory abilities of the shallow layers on deep layers to better filter the noise——>operation: residual operation and recursive operation
- single processing methods: ?
- Data augmentation: horizontal flip, vertical flip and color jittering; Using GAN to construct virtual noisy images is also useful for image denoising
- Transfer learning, graph and neural architecture search
- imporve hardware or camera mechanism
trick:
- compress neural networks,
- reduce the depth or the width of deep neural networks: reduce complexity of these networks
- use small convolutional kernal and group convolution: reduce the number of parameters, accelerating the speed of training
- fusion of dimension reduction methods: PCA and CNN
Complex noisy images: step by step processing, for example, two steps: one recover high resolution image by CNN, another uses a novel CNN to filter the noise of high-resolution image
several as yet unsolved problems
- deeper denoising networkds require more memory resources
- deeper denoising networks is not a stable solution for real noisy image, unpaired noisy image and muliti-degradation tasks
- real noisy images are not easily captured
- difficult to solve unsupervised denoising tasks
- more accurate metrics need to be found for image denoising
6 Conclusion
regular Gaussian noise has achieved great success, while the noise in the real world is complex and irregular
Imporve hardware devices in order to suppress the noise for capturing a high-quality image
noisy images with ground truth