[论文笔记]-SR:Coupled Convolutional Neural Network With Adaptive Response Function

论文标题：Coupled Convolutional Neural Network With Adaptive Response Function Learning for Unsupervised Hyperspectral Super Resolution

Abstract

---

本文提出了一种基于无监督深度学习的融合方法- - HyCoNet - -该方法可以解决HSI - MSI融合中没有先验PSF和SRF信息的问题

HyCoNet consists of three coupled autoencoder nets in which the

HSI and MSI are unmixed into endmembers and abundances based on the linear unmixing model

“Two special convolutional layers are designed to act as a bridge that coordinates with the three autoencoder nets, and the PSF and SRF parameters are learned adaptively in the two convolution layers during the training process.” (Liao 等, 2021, p. 2487)

joint loss function,

Index Terms

---

Adaptive learning, autoencoder, coupled convolutional neural network, hyperspectral image, super-resolution.

I. INTRODUCTION

---

HSI–MSI fusion is similar to the MSI pansharpening process in which a low spatial resolution MSI is fused with a high-resolution panchromatic (PAN) image

现存的高光谱和多光谱融合方法有三种：

• 全色锐化拓展，

• 基于贝叶斯的方法，

• 基于矩阵分解的方法

第一种方法

依赖于光谱重采样

Ry，Gomez尝试用小波来融合MSI和MSI

zhang提出了一种基于三维小波变换的融合方法

chen提出了一种融合HSI和MSI的框架——将HSI划分为多个区域（使用全色锐化方法）

Aiazzi等[ 25 ]提出了一种将光谱响应函数( SRF )作为模型一部分的成分替代融合方法——将原有的复杂数据简化成了SRF

第二种方法

Esiman 提出了一种基于贝叶斯的方法，利用底层光谱内容的随机混合模型实现分辨率增强

wei提出了一种变分的融合方法，将目标图形啊假设在低维空间中，通过优化编码系数和目标来解决融合问题

Sim将融合问题建模为包含两个二次项和一个边缘保持项的凸目标的最小化问题

Akhtar提出了一种非参数贝叶斯系数编码策略，首先推断材料光谱的概率分布，然后计算高光谱图像的稀疏编码

第三种方法

通常假设HSI由一系列纯光谱向量组成，HSI可以被分解为丰度和端元

融合问题被分为低分辨率HSI端元和高分辨率MSI丰度的估计问题

Kawakami提出了将低分辨率HSI和高分辨率RGB图像融合的解混方法

首先，利用解混算法估计低分辨率HSI的基端元，然后将其与高分辨率RGB图像结合，生成最终结果（也就是两个都有所涉及）

Yokoya和Lanaras提出一种交替解混方法估计端元和峰度的耦合NMF（CNMF）

Wycoff将稀疏正则化限制在丰度上——假设每个像素仅有少量端元组成

Akhtar提出了一种基于局部空间结构约束的稀疏表示方法——主要利用贪婪追逐算法挖掘局部斑块先验知识

Yi提出了一种结合光谱解混和空间约束的交互式反馈策略

Tensor-based fusion methods utilizing tensor factorization with sparse constraint or subspace projection have also been proposed [14], [37].

深度学习应用

监督类（无法获得高分辨率HSI——难应用）

• wang

• Han——部分稠密连接网络

• Palsson——3D卷积网络，降低HSI纬度来加速

• Dian——Sylvester方程初始化HSI,然后执行nn来学习HSI映射MSI

• Xie——融合网络，考虑低分辨率图像和低秩

• wang——提出了一种基于迭代和交替优化策略学习盲HSI

• Han——提出了一种HSI-MSI融合的多尺度空间和光谱融合卷积神经网络

本文采用的是非监督分类的方法来解决问题

无监督

QU——稀疏狄利克雷

ZHOU——配准算法和融合算法来处理具有（显著尺度差异和非刚性失真的HSI和MSI图像）

Fu——提出CSR来优化层，学习光谱响应

本文提出的是PSF和SRF未知的情况

需要唯一的信息就是MSI和HSId的光谱覆盖率——很容易从数据提供者所获得

II. PROBLEM FORMULATION

---

In addition, the abundances should be sparse, meaning that each HSI pixel is composed of only a few pure spectral bases.

III. PROPOSED METHOD

---

要解决这些融合问题，关键在于估计空间分辨率丰度矩阵A和光谱基矩阵E

Hr MSI 提供了A高度相关的详细空间上下文信息

Lr HSI提供了与目标光谱基E高度一致的光谱信息

“A. Coupled Autoencoder Network for Image Fusion”

The estimated target HrHSI X is embedded in the HrMSI autoencoder.

The LrMSI autoencoder can be seen at the upper right.

Instead of using the fully connected layer, the convolution layer is used to preserve the spatial structure of the input image cube for easy implementation of the PSF operation.（对于HrMSI不使用全连接层，而是用图像立方体[1*1卷积]保留PSF操作）

PSF具体操作

之所以我们把潜变量Aa h和参数E称为丰度和端元，是因为它们试图通过矩阵乘法重建Z_a

A是丰度，E是端元

对A和E的训练采用了几种技巧（in Section III-B.）

上面的自编码结构和下面的自编码结构类似

“The number of convolution kernels is equals to the number of bands in ̃ Y.” (Liao 等, 2021, p. 2491) 卷积核个数等于" Y "中的波段数。

ThePSF means that a given pixel is a weighted combination of contributions from the pixel and its neighboring pixels [54], [55]

fig 3很清晰的说明了其中的转换关系

“In addition, the spatial degraded version of the HrMSI is equivalent to the spectrally degraded version of the LrHSI shown as “LrMSI-Autoencoder” in Fig. 2.” (Liao 等, 2021, p. 2491) 此外，

Hr MSI的空间降质版本相当于图2中" Lr MSI-Autoencoder "所示的Lr HSI的光谱降质版本。

“B. Joint Loss Function” (Liao 等, 2021, p. 2491)

两个限制

这里没有用到softmax函数，收敛精度低于本文方法

clamp function

the output of the last convolution layer of both the encoder nets and decoder nets to force all the elements of the abundances and reconstruction images into the range [0,1].

本来想用sigmoid 激活函数得到，但是对于网络很难收敛

共享卷积层

也是满足非负特性

K-L散度的使用

HSI应该是由销量纯光谱基构成的，因此A应该是稀疏的。所以使用K-L散度

把几个限制函数都拼到一起得到

我们的目的是按照最优化问题来解决融合问题

IV. EXPERIMENTS

---

文章使用了仿真实验来评估融合质量和方案性能

1. First, the sensitivity of the tradeoff parameters α, β, γ , μ,andν was evaluated.

2. Second, the constraints on the abundances were investigated.

3. Third, we compared the effectiveness of the method for different numbers of endmembers.

4. Fourth, the learned PSF kernels were investigated using different spatially down-sampled kernels.

Finally, the fused images obtained using the different methods were evaluated using both visual and quantitative measures.

A. Experimental Data Set

---

Pavia University, Indian Pines, Washington DC, and University of Houston.

B. Implementation Details

---

利用高斯滤波器来获得LrHSI影像数据，实现空间下采样模拟

In the experiment, the width and height of the Gaussian filter was set equal to the ratio between the high-resolution GSD and the low-resolution GSD.

分辨率通常被定义为地面样本距离(GSD)

利用五个不同的质量来评估

the RMSE, MRAE, and SAM were used in the visual evaluation,

and the PSNR, SAM, and ERGAS were used in the quantitative evaluation.

优化器的选择

“The proposed model was trained using an Adam optimizer [61] with the default parameters β1 = 0.9, β2 = 0.999, and ε = 10−8;” (Liao 等, 2021, p. 2493)

C. Parameters Discussion

---

五大参数

HSI-MSI融合是由自编码网络驱动的，重建误差很关键

first, we set the parameters γ , μ,andν to a fixed value of 1 to evaluate the effect of using different values of the parameters α and β.

In this part of the experiment, we used the Pavia University data to investigate the performance.

we used the Pavia University data to investigate the performance

The best result is achieved for α = β = 10. Therefore, we set α and β to 10 in the later experiments.
we set γ to 100 in the later experiments.

很明显μ敏感，v不敏感

“Accordingly, we set μ and ν to 0.001 in the subsequent experiments.” (Liao 等, 2021, p. 2494) 因此，在后续实验中我们将μ和ν设置为0.001。

消融实验

To investigate the essentiality of the proposed network, as shown in Fig. 5, the ablation study was implemented in the case of the hyperparameters setting mentioned above.

通过消融实验，就可以看出来当缺少某些网络和损失时候的表现性能

As shown in Fig. 5,⊂ means removing certain parts of the network or losses.

就是拆解掉某个模块的时候，比如zb

It can be seen that the performances suddenly drop when removing ̃ Zb.

很明显zb的影响很大

D. Constraint Function for the Abundance

---

因为是丰度的讨论，所以只用讨论到0-1即可

Equation (17) indicates that the sum-to-one property is restricted in the loss function;

文章使用了clamp function但实事上hong用softmax也可以解决这个问题，前面已经交代了这些内容

E. Number of Endmembers

---

讨论了端元数对其的影响，设置了不同的p值来进行实验

在我们的模型中，端元个数p表示了丰度特征尺寸的个数， 也表示共享卷积层的核个数

For convenience, we set p = 100 in all cases when exploring the performance of the proposed model.

F. Learned PSF Kernel

---

原始的数据都是HrHSI,所以利用PSF进行模拟

将原始的数据（Pavia University and Washington DC data）评估了不同的条件下learned kernels

设置了高斯滤波器的标准差分别为0.5,1,2

依据原始数据进行核尺寸大小为4和8

点扩散函数的可视化图

直观地展示了本文方法对任意PSF具有较高的适用性

G. Estimated Abundances

---

“In our network, the abundance is constrained by the sparse and sum-to-one characteristics of the loss function” (Liao 等, 2021, p. 2496)

we next used the Pavia University data to explore the three estimated abundances corresponding to Aa h, Ab h ,andA.The histograms of these abundances are shown in Fig. 9(a)

由下面的图一看显然内容很少，是稀疏的

“The heatmap for Aa h shows that some of the” areas at the edges did not completely satisfy the sum-to-one constraint.(一些综述根本没有达到加起来是1的限制)

H. Comparison With the State of the Art

---

“1) Visual Performance:” (Liao 等, 2021, p. 2498) 1 )视觉表现：

跟随Yokoya的步伐，很多基础的方法都用来比较

包含的方法有

line methods were used for comparison. These included CNMF [32], GSOMP [35], FUSE [64], GLPHS [65], GSA [25], HySure [29], Lanaras’s method (for convenience, we called it ICCV15 because it was published in proceedings of the 2015 International Convention on Computer Vision) [33], MAPSMM [27], SFIM-HS [26], and uSDN [49].

将所有的方法都公平比较

比较不同的方法

SAM代表了每个像素的光谱一致性

“2) Quantitative Performance:” (Liao 等, 2021, p. 2499) 2 )定量性能：

展示了不同方法的具体性能

评价一个每个方法在各个方面上的长处

The results for theIndian Pines data are stable because the GSD for these data is the largest and the land objects are the simplest.

对于华盛顿特区数据所有方法都不太好

“Fig. 14 shows the PSNR for the different bands of the HSI when σ = 0.5, showing the reconstructed spatial quality for each band.” (Liao 等, 2021, p. 2499) 图14给出了σ = 0.5时HSI不同波段的PSNR，显示了每个波段的重构空间质量。

V. CONCLUSION

---

In this article, we proposed a novel unsupervised deep learning method called HyCoNet to solve the HSI and MSI fusion problem for arbitrary PSFs and SRFs.

The experiments that were performed indicated that the proposed method solved the HSI and MSI fusion problem without knowing the prior information of the PSF and SRF, and produced stable and robust fusion results for arbitrary PSFs and SRFs.

致谢

---

致谢里面说了说句的原来和论文作者的代码