from: http://www.vsislab.com/papers/StereoQA/StereoQA.html
Learning Structure of Stereoscopic Image for No-Reference Quality Assessment with Convolutional Neural Network
Wei Zhang1, Chenfei Qu1, Lin Ma2, Jingwei Guan3, and Rui Huang4
1School of Control Science and Engineering, Shandong University
2Huawei Noah's Ark Lab, Hong Kong
3Department of Electronic Engineering, The Chinese University of Hong Kong
4NEC Laboratories China
Abstract
In this paper, we propose to learn the structures of stereoscopic images based on convolutional neural network (CNN) for no-reference quality assessment. Taking image patches from the stereoscopic images as inputs, the proposed CNN can learn the local structures which are sensitive to human perception and representative for perceptual quality evaluation. By stacking multiple convolution and max-pooling layers together, the learned structures in lower convolution layers can be composed and convolved to higher levels to form a fixed-length representation. Multilayer perceptron (MLP) is further employed to summarize the learned representation to a final value to indicate the perceptual quality of the stereo image patch pair. With different inputs, two different CNNs are designed, namely one-column CNN with only the image patch from the difference image as input, and three-column CNN with the image patches from left-view image, right-view image, and difference image as the input. The CNN parameters for stereoscopic images are learned and transferred based on the large number of 2D natural images. With the evaluation on public LIVE phase-I, phase-II, and IVC stereoscopic image databases, the proposed no-reference metric achieves the state-of-the-art performance for quality assessment of stereoscopic images, and is even competitive to existing full-reference quality metrics.
The contributions of this work:
-
CNNs are employed to learn the local structures for stereoscopic image quality assessment. The proposed CNN couples the feature extraction and learning process together to produce the perceptual quality from the image pixel.
-
Two CNNs are designed to learn the image local structures based on different inputs. Difference image are introduced to CNN to assess the image quality. Experimental results on public stereoscopic image datasets show that our proposed CNN model surpasses the state-of-the-art.
-
CNN parameters are pretrained on 2D images and transferred to stereoscopic images, which solves the problem of the lacking of stereoscopic image, and improves the corresponding performances.
Convolutional Neural Network for Stereoscopic Image Quality Assessment
The stereoscopic images differ from the 2D natural images, as the left and right views together can provide depth perception. Therefore, perceptual evaluation of the stereoscopic image needs to consider the information from both the left and right views. It is demonstrated that the difference image between the left view and right view is more important than the left and right views for quality assessment.
![](https://i-blog.csdnimg.cn/blog_migrate/f42c810e74be715d7365fb9a355d3eec.png)
Proposed three-column CNN:
The three-column CNN consists three one-column CNNs, which takes full consideration of not only left view and right view in stereoscopic images, but also the difference image obtaining from the stereoscopic image. The difference image implicitly contains the disparity and depth information, which is proved to be very important for the stereoscopic image quality perception.
Each column in the three-column CNN has two layers of convolution and max-pooling. The three identical CNNs are used to learn the structures from the left view, right view, and difference image, respectively. With two layers of convolution and max-pooling processes, three different image patches are represented as three different vectors, which are expected to contain the structures of stereoscopic images from different viewpoints. These three vectors are concatenated together and fed into the upper multilayer perceptron (MLP) to generate the quality score.
Experimental Results
SROCC of IQA metrics on LIVE phase-I dataset
SROCC of IQA metrics on LIVE phase-II dataset (with symmetric distortion)
SROCC of IQA metrics on LIVE phase-II dataset (with asymmetric distortion)
LCC and SROCC of IQA metrics on IVC dataset
Reference
Benoit et al. A. Benoit, P. Le Callet, P. Campisi, and R. Cousseau, "Quality assessment of stereoscopic images", EURASIP Journal on Image and Video Processing, 2009. [ Full Text]You et al. J. You, L. Xing, A. Perkis, and X. Wang, "Perceptual quality assessment for stereoscopic images based on 2D image quality metrics and disparity analysis", International Workshop on Video Processing and Quality Metrics for Consumer Electronics, 2010. [ Full Text]Gorley and Holliman. P. Gorley and N. Holliman, "Stereoscopic image quality metrics and compression", Proc. SPIE, 2008. [ Full Text]MS-SSIM M. J. Chen, D. K. Su, C. C. Kwon, L. K. Cormack, and A. C. Bovik, "Full-reference quality assessment of stereopairs accounting for rivalry", Proc. Asilomar Conference on Signals, Systems and Computers, 2012. [ Full Text]Hewage and Martini C. Hewage, S. T. Worrall, S. Dogan, and A. M. Kondoz, "Prediction of stereoscopic video quality using objective quality models of 2-D video", Electronics Letters, 2008. [ Full Text]Wang et al. Z. Wang, E. P. Simoncelli, and A. C. Bovik, "Multiscale structural similarity for image quality assessment", Proc. Asilomar Conference on Signals, Systems, and Computers, 2003. [ Full Text]Ma et al. L. Ma, X. Wang, Q. Liu, and K. N. Ngan, "Reorganized DCT-based image representation for reduced reference stereoscopic image quality assessment", Neurocomputing, 2016. [accept]Akhter et al. R. Akhter, J. Baltes, Z. M. Parvez Sazzad, and Y. Horita, "No reference stereoscopic image quality assessment", Proc. SPIE, 2010. [ Full Text]Shao et al. F. Shao, W. Lin, S. Wang, G. Jiang, and M. Yu, "Blind image quality assessment for stereoscopic images using binocular guided quality lookup and visual codebook", IEEE Transactions on Broadcasting, 2015. [ Full Text]Ryu and Sohn S. Ryu, D. H. Kim, and K. Sohn, "Stereoscopic image quality metric based on binocular perception model", ICIP, 2012. [ Full Text]Chen et al. M. J. Chen, L. K. Cormack, A. C. Bovik, "No-reference quality assessment of natural stereopairs", TIP, 2013. [ Full Text]SSIM Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity", TIP, 2004. [ Full Text]FSIM L. Zhang, D. Zhang, X. Mou, and D. Zhang, "FSIM: A feature similarity index for image quality assessment", TIP, 2011. [ Full Text]GSMD W. Xue, L. Zhang, X. Mou, and A. C. Bovik, "Gradient magnitude similarity deviation: A highly efficient perceptual image quality index", TIP, 2014. [ Full Text]Bensalma et al. R. Bensalma and M.-C. Larabi, "A perceptual metric for stereoscopic image quality assessment based on the binocular energy", Multidimensional Systems and Signal Processing, 2013. [ Full Text]Chen et al. M. J. Chen, D. K. Su, C. C. Kwon, L. K. Cormack, and A. C. Bovik, "Full-reference quality assessment of stereopairs accounting for rivalry", Proc. Asilomar Conference on Signals, Systems and Computers, 2012. [ Full Text]Carnec et al. M. Carnec, P. Le Callet, and D. Barba, "An image quality assessment method based on perception of structural information", IEEE International Conference on Image Processing, 2003. [ Full Text]Skeikh et al. H. R. Sheikh, A. C. Bovik, and G. de Veciana, "An information idelity criterion for image quality assessment using natural scene statistics", TIP, 2005. [ Full Text]Venkata et al. N. Damera-Venkata, T. D. Kite, W. S. Geisler, B. L. Evans, and A. C. Bovik, "Image quality assessment based on a degradation model", TIP, 2000. [ Full Text]Ryu et al. S. Ryu, D. H. Kim, and K. Sohn, "Stereoscopic image quality metric based on binocular perception model", IEEE International Conference on Image Processing, 2012. [ Full Text]Campisi et al. P. Campisi, P. Le Callet, and E. Marini, "Stereoscopic images quality assessment", European Signal Processing Conference, 2007. [ Full Text]
Contact Me
If you have any questions, please feel free to contact Prof. Zhang (info@vsislab.com).Update: Apr. 14, 2016