Image Restoration Using Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections

##一、本文的主要贡献
1. 卷积和反卷积对称网络结构的提出

A very deep network architecture, which consists of a chain of symmetric convolutional and deconvolutional layers, for image restoration is proposed in this paper. The convolutional layers act as the feature extractor which encode the primary components of image contents while eliminating the corruption. The deconvolutional layers then decode the image abstraction to recover the image content details.

2. 卷积和反卷积使用跳层结构相连(模仿残差网络)------有助于恢复图像细节和取得更好的局部最优值

when the network goes deeper, as mentioned above, image details can be lost, making deconvolution weaker in recovering them.

We propose to add skip connections between corresponding convolutional and de-convolutional layers. These skip connections help to back-propagate the gradients to bottom layers and pass image details to the top layers, making training of the end-to-end mapping more easier and effective, and thus achieve performance improvement while the network going deeper.

The passed convolutional feature maps are summed to the deconvolutional feature maps element-wise, and passed to the next layer after rectification(RELU).

Skip connections are passed every two convolutional layers to their mirrored deconvolutional layers.

3. 使用单一模型去处理图像不同程度的噪音或损坏

Relying on the large capacity and fitting ability of our very deep network, we propose to handle different level of noises/corruption using a single model

---

##二、细节解读

1. 训练细节

• **Denoising(去噪)**使用的是 gray-scale image，**super-resolution(超分辨率)**使用的是图片的 luminance channel(亮度通道) 。
• For each image, patches of size $50\ast 50$ are sampled as ground truth.
• For denoising, we add additive Gaussian noise to the patches multiple times to generate a large training set (about 0.5M).
• For super-resolution, we first down-sample a patch and then up-sample it to its original size, obtaining a low-resolution version as the input of the network.

% 设置尺度缩放倍数
scale = 4;

% work on illuminance
image = imread('lena.bmp');
if size(image,3) > 1
    image = rgb2ycbcr(image);
    image = image(:,:,1);                 % 取出亮度这一维的图像数据
end

% generate low-resolution image and run super-resolution
image = modcrop(image, scale);            % 保证图像(image)能够整除尺度(scale)
im_lre = imresize(imresize(single(image)/255,1/scale,'bicubic'),scale,'bicubic');  % 使用双三线性插值先缩小图像后放大图像，得到低分辨率的图片  
im_rec = netforward(im_lre,scale,'super-resolution'); % 调用训练好的模型，进行前向运算

% compute psnr and ssim
PSNR_lre = csnr(im_lre*255,image,0,0)
SSIM_lre = cal_ssim(im_lre*255,image,0,0)

PSNR_rec = csnr(im_rec*255,image,0,0)
SSIM_rec = cal_ssim(im_rec*255,image,0,0)

% show results
imwrite(image, 'image.png');
imwrite(uint8(im_lre*255), 'im_lre.png');
imwrite(uint8(im_rec*255), 'im_rec.png');

2. 测试细节

• To achieve more smooth results, we propose to process a corrupted image on multiple orientations.
• Different from segmentation, the filter kernels in our network only eliminate the corruptions, which is not sensitive to the orientation of image contents. Therefore, we can rotate and mirror flip the kernels and perform forward multiple times, and then average the output to get a more smooth image。

function [img_out] = netforward(img, param, task)

[wid,hei,chn] = size(img);
caffe.reset_all();
caffe.set_mode_cpu();

model = ['model/' task '/' num2str(param) '.caffemodel'];  % model/super_resolution/4.caffemodel 

if strcmp(task,'debluring') || strcmp(task,'inpainting')
    net = caffe.Net('model/REDNet_ch3.prototxt',model,'test');            % 三通道
else
    net = caffe.Net('model/REDNet_ch1.prototxt',model,'test');            % 一通道
end

% 逆时针旋转图片四次和先翻转后逆时针旋转图片四次，然后求八次前向运算后的平均值。
img_out = zeros(wid, hei, chn,8);
for i = 1 : 4
    output = net.forward({rot90(img,i-1)});
    img_out(:,:,:,i) = rot90(output{1}, 5-i) + img;
end
for i = 5 : 8
    output = net.forward({rot90(fliplr(img),i-5)});
    img_out(:,:,:,i) = fliplr(rot90(output{1}, 9-i)) + img;
end

img_out = mean(img_out,4);    % 取第4维数据的平均值，即求经过旋转和翻转的八张图片(前向运算后)的平均值


end

---

##三、评价指标

---

##四、参考文献
Mao, Xiao-Jiao, Chunhua Shen, and Yu-Bin Yang. “Image restoration using convolutional auto-encoders with symmetric skip connections.” arXiv preprint arXiv:1606.08921 (2016).