demoireing

Mop Moire Patterns Uing MopNet(MopNet,Bin He,2019ICCV)

文章指出DMCNN对于非摄屏图像（自然界中的频率条纹图像）的表现性能急剧下降，于是延续了Sun的工作，把摩尔纹的属性类别考虑了进来。" All core components of MopNet are specially designed for unique properties of moire patterns, including the multi-scale feature aggregation to address complex frequency, the channel-wise target edge predictor to exploit imbalanced magnitude among colour channels, and the attribute-aware classifiers to characterize the diverse appearance for better modelling Moir´e patterns."总的来说，作者考虑了频率的分布、边缘强度、摩尔纹属性类别。

1.Introduction

1. We propose a property oriented learning framework for moire pattern removal. With discovered properties of moir´e patterns from multiple aspects including frequency distribution, edge information and appearance attributes, our framework contributes to the comprehensive modelling of moire patterns.
2. We propose specific learning schemes to resolve these useful properties. Particularly, we come up with the multi-scale feature aggregation, channel-wise edge predictor, and attribute-aware classifier, to deal with issues of the complex frequency distribution, imbalanced edge intensities among colour channels, as well as appearance diversity of moir´e patterns, respectively.
3. We have achieved cleaner moir´e removal in the benchmark datasets including those challenging low frequency moire patterns. Our approach may better preserve the structure of target images, which outperforms state-of-the-art methods in terms of numerical metrics and subjective visual quality. Moreover, we
   make additional annotation of attributes over benchmark datasets, which is useful to improve the performance via the property oriented learning framework.

2.Related Work

3.Methodology

3.1.Multi-scale feature aggregation

Sun对于这块的设计采取了多个平行分支(different resolution)来处理multi-scale feature，并进行简单的线性叠加。He对于这块的设计同样采取了多个分支来处理不同频率段的特征，但是采用了SE block进行特征层的叠加，这相当于对特征通道赋予权重，强调了比较重要的频率段分支。

3.2.Channel-wise target edge predictor

摩尔纹的形状伴随着显著的边缘量级变化，同时在不同颜色通道中显现出不平衡的特点，于是作者在网络结构设计中加入了边缘线索。输入为受污染的图像以及在各个通道上进行sobel kernel卷积图，输出为干净图像的边缘图，并把这个边缘图加进整个网络设计中，这样可以起到保护微细结构、防止过度平滑的效果。

同时，在这部分的设计中加入了non-local blocks，“non-local blocks are introduced to assist the predictor in capturing semantic edges with weaker gradient. For each position in a feature map,the non-local block computes correlation weights between features at that position and the rest, then obtain the response as a weighted sum of the features at all positions. The correlation matrix for a point on a weak edge captures long-distance dependencies, which assists the weak edge to obtain a stronger response by strengthening features at strong edges in weighted summation.”

3.3Attributes-aware moire pattern classifier

摩尔纹具有复杂的多样性，作者对摩尔纹进行了细致的观察，引入dominant frequency、colour、shape来对摩尔纹进行描述，这种描述信息在一定程度上指引了学习进程。

“The frequency label helps to specially deal with the low frequency moir´e patterns whose energy is mixed with the image content to avoid over-smoothing the target image when incorrectly treated as high frequency patterns. The colour label emphasizes moir´e patterns consisting of more than one colour, which are more complicated for removal by mingling with the original colour of the target image than single colour ones. The shape label draws attention to curved moir´e patterns which have more complex curvatures than straight stripes. Hence, these three categorized descriptions of pattern attributes would benefit learning of diverse patterns, which are expected to improve the capability of generalization to patterns unseen in training data.”

3.4.objective function

对于网络中的分类问题，作者采用binary cross entropy loss for each attribute，并且为了提高视觉质量，设计的损失函数同时从像素级别和高维特征级别考虑。

4.Experiments

4.1. Quantitative evaluation

DMCNN相比U-Net能取得更好的效果，但PSNR值较低(博主很质疑PSNR这个指标是否适合作为去摩尔纹的评价指标)，而Mop-Net由于考虑了边缘和类别属性，超越现存方法并取得更好的视觉效果，同时获得更高的PSNR和SSIM值(???)。

4.2.Qualitative evaluation

DMCNN对于低频摩尔纹图像效果较差，容易遗留coloured spots or stripes。相比之下，Mop-Net由于考虑了不同分支特征的整合方式，对低频的处理更加高效，添加的边缘信息有效保护细微结构、防止过度平滑效果。

4.3.Ablation study

作者进行消融实验分别验证了各个模块的效果，不同尺度的特征整合策略有效去除了低频摩尔纹，边缘信息保护了图像细微结构，类别属性有效去除了复杂条纹。

同时探索了channel-wise or gray-wise edge map的边缘预测模块的效果(with or without non-local blocks)

5.Discussions

Moire patterns in texture images.作者在此部分探索了网络的泛化性能，对于自然界未见过的真实图像(非摄屏)。

High resolution moire images.因为网络训练是采用跟DMCNN一样的设置，图像采用256X256大小，作者的future work是考虑把超分辨与去摩尔纹结合起来实现端对端的效果。

6.Conclusion

"We proposed MopNet, consisting of a multi-scale aggregated, edge-guided, and pattern attribute-aware network to “mop” moir´e patterns from images. It exploits and specially deals with the properties of moir´e patterns including complex frequency distribution, imbalanced intensities of edges among RGB channels and varying appearance attributes."由于摩尔纹形成还没有一个清晰的物理模型来描述，作者的未来工作便是想办法探讨formation mechanism。

Limitations.去摩尔纹失败的例子，图12由于复杂和不规则的纹理，致使边缘预测失败，导致了摩尔纹去除的不完全效果。