论文笔记：Blind Super-Resolution With Iterative Kernel Correction

论文：Blind Super-Resolution With Iterative Kernel Correction
论文看得比较粗略，所以可能存在理解错误的地方，请指正（另外请忽略我糟糕的英语）

Movtivation

The point is that different kernel will generate different artifact-texture in the result. So you should choose the right kernel. For example, use Gaussian kernel with kernel width ${\sigma }_{LR}$, the SR results show unnatural ringing artifacts when ${\sigma }_{SR}>{\sigma }_{LR}$ and over-smoothing on the other side.

Estimate the kernel

A straightforward method is to adopt a function that estimates kernel from the LR image. Let
$$k^{\prime }=\mathcal{P}(I^{LR})$$
Then we can optimize the function by minimizing the $L_2$ distance
$${\theta }_{\mathcal{P}}=\underset{{\theta }_{\mathcal{P}}}{\mathrm{argmin}}||k-\mathcal{P}(I^{LR};{\theta }_{\mathcal{P}})|{|}_2^2$$
But accurate estimation of kernel is impossible as the problem is ill-posed. So they try to find a way to correct the estimation

Correct the kernel

The idea is to adopt the intermediate SR results. Let $\mathcal{C}$ be the corrector function, then
$${\theta }_{\mathcal{C}}=\underset{{\theta }_{\mathcal{C}}}{\mathrm{argmin}}||k-(\mathcal{C}(I^{SR};{\theta }_{\mathcal{C}})+k^{\prime })|{|}_2^2$$
To avoid over- or under-fitting, a smaller correction steps is used to refine the kernel until it reaches ground truth.

Method

Let $\mathcal{F}$ be a SR model, $\mathcal{P}$ is a kernel predictor and $\mathcal{C}$ is a corrector. You can use PCA to reduce the dimensionality of the kernel space. The kernel after the dimension reduction is denoted by $h$ where $h=Mk$, $M$ is the dimension reduction matrix. An initial estimation $h_0$ is given by the predictor $h_0=\mathcal{P}(I^{LR})$ and the first SR result is $I_0^{SR}=\mathcal{F}(I^{LR},h_0)$, Then the iterative kernel correction algorithm can be written as
$$\begin{array}{rlc}\Delta h_i& =& \mathcal{C}(I^{SR},h_{i-1})\\ h_i& =& h_{i-1}+\Delta h_i\\ I_i^{SR}& =& \mathcal{F}(I^{L}R,h_i)\end{array}$$
After $t$ iterations, the $I_t^{SR}$ is the final result of IKC.

Network Architecture of SR model

The SR method for multiple blur kerenls, SRMD have two problems.

1. The kernel maps do not actually contain the information of the image.
2. The influence of kernel information is only considered at the first layer.

So SFTMD is proposed which using spatial feature transform layer

Use SRResNet as the backbone (of cause you can change it) and then employ the SFT layer to provide the affine transformation for the feature maps $F$ conditioned on the kernel maps $\mathcal{H}$ by a scaling and shifting operation:
$$\mathrm{S}\mathrm{F}\mathrm{T}(F,\mathcal{H})=\gamma \odot F+\beta$$
The kernel maps $\mathcal{H}$ is stretched by $h$, where all the elements of the $i$-th map are equal to the $i$-th element of $h$.

Network Architecture of $\mathcal{P}$ and $\mathcal{C}$

Experiments

Always best.

Comprehension

1. How can I train the predictor $\mathcal{P}$
   Although the paper said that it is an ill-posed problem, then they get trained model. Over-fit or under-fit? What’s the loss curve like? and When should I stop training?
2. Spatially uniform
   How? The paper said it is different from its application in semantic super resolution. Just employ the transformation characteristic of SFT layers.
   So I don’t understand that if the segmentation information can provide spatial variability?
3. IKC With PCA is best?
   Why? As the paper said that the PCA can provides a feature representation and IKC learns the relationship between the SR images and features rather than the Gaussian kernel. But why can’t IKC learn features from kernel?