超分辨率方法的比较——插值、重构、学习

1. Interpolation-based methods        2. Refactoring-based methods        3. Learning-based methods

文章目录

前言

一、基本介绍

1、基于插值的方法

2、基于重构的方法

3、基于学习的方法

二、方法比较

---

前言

参考论文：Remote Sensing | Free Full-Text | Improved SRGAN for Remote Sensing Image Super-Resolution Across Locations and SensorsDetailed and accurate information on the spatial variation of land cover and land use is a critical component of local ecology and environmental research. For these tasks, high spatial resolution images are required. Considering the trade-off between high spatial and high temporal resolution in remote sensing images, many learning-based models (e.g., Convolutional neural network, sparse coding, Bayesian network) have been established to improve the spatial resolution of coarse images in both the computer vision and remote sensing fields. However, data for training and testing in these learning-based methods are usually limited to a certain location and specific sensor, resulting in the limited ability to generalize the model across locations and sensors. Recently, generative adversarial nets (GANs), a new learning model from the deep learning field, show many advantages for capturing high-dimensional nonlinear features over large samples. In this study, we test whether the GAN method can improve the generalization ability across locations and sensors with some modification to accomplish the idea “training once, apply to everywhere and different sensors” for remote sensing images. This work is based on super-resolution generative adversarial nets (SRGANs), where we modify the loss function and the structure of the network of SRGANs and propose the improved SRGAN (ISRGAN), which makes model training more stable and enhances the generalization ability across locations and sensors. In the experiment, the training and testing data were collected from two sensors (Landsat 8 OLI and Chinese GF 1) from different locations (Guangdong and Xinjiang in China). For the cross-location test, the model was trained in Guangdong with the Chinese GF 1 (8 m) data to be tested with the GF 1 data in Xinjiang. For the cross-sensor test, the same model training in Guangdong with GF 1 was tested in Landsat 8 OLI images in Xinjiang. The proposed method was compared with the neighbor-embedding (NE) method, the sparse representation method (SCSR), and the SRGAN. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were chosen for the quantitive assessment. The results showed that the ISRGAN is superior to the NE (PSNR: 30.999, SSIM: 0.944) and SCSR (PSNR: 29.423, SSIM: 0.876) methods, and the SRGAN (PSNR: 31.378, SSIM: 0.952), with the PSNR = 35.816 and SSIM = 0.988 in the cross-location test. A similar result was seen in the cross-sensor test. The ISRGAN had the best result (PSNR: 38.092, SSIM: 0.988) compared to the NE (PSNR: 35.000, SSIM: 0.982) and SCSR (PSNR: 33.639, SSIM: 0.965) methods, and the SRGAN (PSNR: 32.820, SSIM: 0.949). Meanwhile, we also tested the accuracy improvement for land cover classification before and after super-resolution by the ISRGAN. The results show that the accuracy of land cover classification after super-resolution was significantly improved, in particular, the impervious surface class (the road and buildings with high-resolution texture) improved by 15%.https://www.mdpi.com/2072-4292/12/8/1263

---

一、基本介绍

图像超分辨率模型的基本假设是，如果低空间分辨率图像遵循与创建低空间分辨率图像相同的重采样过程，则低空间分辨率图像中的缺失细节可以被重建或从其他高空间分辨率图像中学习。基于这一假设，近十年来，人们一直致力于精确预测点扩散函数(point spread function, PSF)，它代表了形成低分辨率像素的混合过程。主要有三组方法:1)基于插值的方法，2)基于重构的方法，3)基于学习的方法。

1、基于插值的方法

首先，基于插值的方法是基于一定的数学策略，从相关点计算出待恢复目标点的像素值，具有低复杂度和高效率的特点。但结果图像的边缘效应明显，插值过程中没有产生新的信息，无法恢复图像的细节。

2、基于重构的方法

其次，基于重构的方法对成像过程进行建模，整合来自同一场景的不同信息，获得高质量的重构结果。通常，这些方法以时间差异换取空间分辨率的提高，这通常需要预先注册和大量的计算。

3、基于学习的方法

第三，基于学习的方法[12-20]通过确定重建方法的分辨率提高倍数，克服了困难的局限性，可以面向单幅图像，这是目前超分辨率重建的主要发展方向。在这一类中，常用的方法有近邻嵌入方法(NE)、稀疏表示方法(SCSR)和深度学习方法。

二、方法比较

方法类型	基本假设	代表模型	优点	缺点
基于插值的方法	当前像素的值可以用附近的像素表示	The nearest neighbor interpolation	低复杂度，高效率	没有图像纹理细节可以预测，通常使图像看起来更平滑
		The bilinear interpolation
		The bicubic interpolation
基于重构的方法	通过图像可以恢复其物理性质和特征点扩展函数(PSF)的这些规则可以进一步应用于细节恢复	Joint MAP registration	将同一场景中的不同信息进行融合，获得高质量的信息重建结果	需要预注册，计算量大
		Sparse regression and natural image prior
		Kernel regression
		PSF deconvolution
基于学习的方法	通过对大量图像样本的学习，可以得到点扩展函数	Neighbor-embedding (NE)	当训练样本更接近目标图像时获得更好的性能，当涉及大量样本时可以获得更高的PSNR	这非常耗时，需要大量的训练数据集，并且通常限制了跨数据集的模型泛化能力
		Convolutional neural network (SRCNN)
		Bayesian networks
		Kernel-based methods
		SVM-based methods
		Sparse representation (SCSR)
		SRGAN