A Survey on Rain Removal from Video and Single Image

Hong Wang, Yichen Wu, Minghan Li, Qian Zhao, and Deyu Meng

[Arxiv]

Citation

@article{WangA,

title={A Survey on Rain Removal from Video and Single Image},

author={Wang, Hong and Wu, Yichen and Li, Minghan and Zhao, Qian and Meng, Deyu},

journal={arXiv preprint arXiv:1909.08326},

year={2019}

}

Physical Properties of Raindrops

• Gemometric Property
  • Terminal velocity of raindrops aloft (JAMC1969), Foote et al [PDF]
  • A new model for the equilibrium shape of raindrops (JAS1987), Beard et al. [PDF]
• Brightness Property
  • Photometric model of a rain drop (Technical Report, Columbia University2004), Garg et al [PDF]
  • Vision and Rain (IJCV2007), Garg et al [Project][PDF]
• Chromatic Property
  • Rain removal in video by combining temporal and chromatic properties (ICME2006), Zhang et al [Project][PDF]
• Spatial and Temporal Propety
  • Simulation of rain in videos (TAS2003), Starik et al [PDF]
  • Pixel based temporal analysis using chromatic property for removing rain from videos (CIC2009), Liu et al [PDF]

Video Deraining Methods

• Time Domain

  • Detection and removal of rain from videos (CVPR2004), Garg et al [Project][PDF]
  • When does camera see rain? (ICCV2005), Garg et al [Project][PDF]
  • Rain removal using kalman filter in video (ICSMA2008), Park et al [PDF]
  • Using the shape characteristics of rain to identify and remove rain from video (S+SSPR2008), Brewer et al [PDF]
  • The application of histogram on rain detection in video (JCIS2008), Zhao et al [PDF]
  • Rain or snow detection in image sequences through use of a histogram of orientation of streaks (IJCV2011), Bossu et al [PDF]
  • A probabilistic approach for detection and removal of rain from videos (IETE JR2011), Tripathi et al [PDF]
  • Video post processing: low latency spatiotemporal approach for detection and removal of rain (IET IP2012), Tripathi et al [PDF]
  • Removal of rain from videos: a review (SIVP2014), Tripathi et al [PDF]
  • Stereo video deraining and desnowing based on spatiotemporal frame warping (ICIP2014), Kim et al [PDF]

• Frequency Domain

  • Spatio-temporal frequency analysis for removing rain and snow from videos (PACV2007), Barnum et al [Project] [PDF]
  • Analysis of rain and snow in frequency space (IJCV2010), Barnum et al [Project] [PDF]

• Low Rank and Sparsity

  • A generalized low-rank appearance model for spatio-temporally correlated rain streaks (ICCV2013), Chen et al [PDF]
  • A rain pixel recovery algorithm for videos with highly dynamic scenes (TIP2013), Chen et al [PDF]
  • Video deraining and desnowing using temporal correlation and low-rank matrix completion (TIP2015), Kim et al [PDF] [Code]
  • Adherent raindrop modeling, detection and removal in video (TPAMI2016), You et al. [Project] [PDF]
  • Video desnowing and deraining based on matrix decomposition (CVPR2017), Ren et al [PDF] [Code]
  • A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors (CVPR2017), Jiang et al [PDF]
  • Should We encode rain streaks in video as deterministic or stochastic? (ICCV2017), Wei et al [PDF] [Code]
  • A directional global sparse model for single image rain removal (AMM2018), Deng et al [PDF] [Code]
  • Video rain streak removal by multiscale convolutional sparse coding (CVPR2018), Li et al [Project] [PDF] [Code]
  • Fastderain: A novel video rain streak removal method using directional gradient priors (TIP2019), Jiang et al [PDF] [Code]

• Deep Learning

  • Robust video content alignment and compensation for rain removal in a cnn framework (CVPR2018), Chen et al [PDF] [Code]
  • Erase or fill? deep joint recurrent rain removal and reconstruction in videos (CVPR2018), Liu et al. [Project][PDF] [Code]
  • D3R-Net: dynamic routing residue recurrent network for video rain removal (TIP2018), Liu et al. [PDF]

Single Image Deraining Methods

• Filter based methods

  • Guided image filtering (ECCV2010), He et al. [Project] [PDF] [Code]
  • Removing rain and snow in a single image using guided filter (CSAE2012), Xu et al. [PDF]
  • An improved guidance image based method to remove rain and snow in a single image (CIS2012), Xu et al. [PDF]
  • Single-image deraining using an adaptive nonlocal means filter (ICIP2013), Kim et al. [PDF]
  • Single-image-based rain and snow removal using multi-guided filter (NIPS2013), Zheng et al. [PDF]
  • Single image rain and snow removal via guided L0 smoothing filter (Multimedia Tools and Application2016), Ding et al. [PDF]

• Prior based methods

  • Automatic single-image-based rain streaks removal via image decomposition (TIP2012), Kang et al [PDF] [Code]
  • Self-learning-based rain streak removal for image/video (ISCS2012), Kang et al. [PDF]
  • Single-frame-based rain removal via image decomposition (ICA2013), Fu et al. [PDF]
  • Exploiting image structural similarity for single image rain removal (ICIP2014), Sun et al. [PDF]
  • Visual depth guided color image rain streaks removal using sparse coding (TCSVT2014), Chen et al [PDF]
  • Removing rain from a single image via discriminative sparse coding (ICCV2015), Luo et al [PDF] [Code] pwd: d229
  • Rain streak removal using layer priors (CVPR2016), Li et al [PDF] [Code]
  • Single image rain streak decomposition using layer priors (TIP2017), Li et al [PDF]
  • Error-optimized dparse representation for single image rain removal (IEEE TIE2017), Chen et al [PDF]
  • A hierarchical approach for rain or snow removing in a single color image (TIP2017), Wang et al. [PDF]
  • Joint bi-layer optimization for single-image rain streak removal (ICCV2017), Zhu et al. [PDF]
  • Convolutional sparse and low-rank codingbased rain streak removal (WCACV2017), Zhang et al [PDF]
  • Joint convolutional analysis and synthesis sparse representation for single image layer separation (CVPR2017), Gu et al [PDF] [Code]
  • Single image deraining via decorrelating the rain streaks and background scene in gradient domain (PR2018)， Du et al [PDF]

• Deep Learning

  • Restoring an image taken through a window covered with dirt or rain (ICCV2013), Eigen et al. [Project] [PDF] [Code]
  • Attentive generative adversarial network for raindrop removal from a single image (CVPR2018), Qian et al [Project] [PDF]
  • Clearing the skies: A deep network architecture for single-image rain streaks removal (TIP2017), Fu et al. [Project] [PDF] [Code]
  • Removing rain from single images via a deep detail network (CVPR2017), Fu et al. [Project] [PDF] [Code]
  • Image de-raining using a conditional generative adversarial network (Arxiv2017), Zhang et al [PDF] [Code]
  • Deep joint rain detection and removal from a single image (CVPR2017), Yang et al.[Project] [PDF] [Code]
  • Residual guide feature fusion network for single image deraining (ACMMM2018), Fan et al. [Project] [PDF]
  • Fast single image rain removal via a deep decomposition-composition network (Arxiv2018), Li et al [Project]) [PDF] [Code]
  • Density-aware single image de-raining using a multi-stream dense network (CVPR2018), Zhang et al [PDF] [Code]
  • Recurrent squeeze-and-excitation context aggregation net for single image deraining (ECCV2018), Li et al. [PDF] [Code]
  • Rain streak removal for single image via kernel guided cnn (Arxiv2018), Wang et al [PDF]
  • Physics-based generative adversarial models for image restoration and beyond (Arxiv2018), Pan et al [PDF]
  • Learning dual convolutional neural networks for low-level vision (CVPR2018), Pan et al [Project] [PDF] [Code]
  • Non-locally enhanced encoder-decoder network for single image de-raining (ACMMM2018), Li et al [PDF] [Code]
  • Unsupervised single image deraining with self-supervised constraints (ICIP2019), Jin et al [PDF]
  • Progressive image deraining networks: A better and simpler baseline (CVPR2019), Ren et al [PDF] [Code]
  • Spatial attentive single-image deraining with a high quality real rain dataset (CVPR2019), Wang et al [Project] [PDF] [Code]
  • Lightweight pyramid networks for image deraining (TNNLS2019), Fu et al [PDF] [Code]
  • Joint rain detection and removal from a single image with contextualized deep networks (TPAMI2019), Yang et al [PDF] [Code]

• Joint Model-driven and Data-driven

  • Deep Layer Prior Optimization for Single Image Rain Streaks Removal (ICASSP2018), Liu et al [PDF]
  • Learning bilevel layer priors for single image rain streaks removal (SP Letters 2018), Mu et al [PDF]
  • Semi-supervised transfer learning for image rain removal (CVPR2019), Wei et al [PDF] [Code]

Datasets and the Usage

• Video

  • Synthetic Datasets: highway and park.
  • Real Datasets: compfinal and night. Please download from [Baidu Netdisk] provided by Li Minghan.

• Single Image

  • Synthetic Datasets: RainTrainL/Rain100L, RainTrainH/Rain100H, Rain12600/Rain1400, and Rain12. Please download from [Baidu Netdisk] provided by Ren Dongwei.
  • Real Datasets: Please download SPA-Data from [Baidu Netdisk, key: 4fwo] provided by Wang Tianyu and Internet-Data from the link provided by Weiwei .

*We note that:

i. RainTrainL/Rain100L and RainTrainH/Rain100H are synthesized by Yang Wenhan. Rain12600/Rain1400 is from Fu Xueyang and Rain12 is from Li Yu.

ii. In video experiment, the rain-removed results of the deep learning method are provided by the author Yang Wenhan. Really thanks!

iii. In single image experiment, we seperately retrain all the recent state-of-the-art methods via the three training datasets: RainTrainL(200 input/clean image pairs), RainTrainH(1800 pairs), and Rain12600(12600 pairs), and then evaluate their rain removal performance based on the correponding test datasets: Rain100L(100 pairs), Rain100H(100 pairs), and Rain1400(1400 pairs). Besides, the trained model obtained by RainTrainL is adpoted to predict rain-removed results of Rain12(12 pairs). Moreover, we utilize the Internet-Data(147 input images) and SPA-Data(1000 pairs) to compare the generalization ability.

iiii. In single image experiment, when training the semi-supervised method--SIRR, we always utilize Internet-Data as unsupervised samples.

Image Quality Metrics

• PSNR (Peak Signal-to-Noise Ratio) [PDF] [Matlab Code] [Python Code]
• SSIM (Structural Similarity) [PDF] [Matlab Code] [Python Code]
• VIF (Visual Quality) [PDF] [Matlab Code]
• FSIM (Feature Similarity) [PDF] [Matlab Code])

*Please note that all quantitative results are computed based on Y channel.

Contact

If you have any question, please feel free to concat Hong Wang (Email: hongwang01@stu.xjtu.edu.cn).