Spatial-Angular Attention Network for Light Field Reconstruction【论文阅读笔记】

作者

Gaochang Wu, Yebin Liu, Lu Fang, Tianyou Chai

相关链接

• Paper:https://arxiv.org/pdf/2007.02252.pdf
• Code:https://github.com/GaochangWu/SAAN
• Video:https://www.bilibili.com/video/av498739331/

需要补的知识点/论文

• non-local attention machanism
• U-net

Abstract

I.Introduction

• non-local attention machanism？
• multi-scale reconstruction structure?
• U-net？
  an encoder-decoder structure with skip connections
• 3D auto-encoder

3 contributions

• A spatial-angular attention module(SAAM) that perceives correspondences non-locally in the epipolar plane;(non-local attention machanism)
• A multi-scale reconstruction structure for efficiently performing the non-locall attention in the low spatial scale while also preserving the high frequencies;(multi-scale reconstruction structure;U-net)
• A spatial-angular perceptual loss specifically designed for the high-dimensional light filed data.(through 3D auto-encoder)

II.Related Work

A.Light Filed Resconstruction

• Depth image-based view synthesis
• Reconstruction without explicit depth

B.Attention Machanism

• A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS, 2017, pp. 5998–6008.
• X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7794–7803.
• H. Zhang, I. Goodfellow, D. N. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” arXiv: Machine Learning, 2018.

III.Problem Analysis

• EPI像素的斜率越陡，视差越小

• 大的感受野会有更好的重建效果
  

• 但受参数量限制，没法一直加大卷积核大小，作者提出使用non-local perception（1.受non-local attention mechanism的启发，提出spatial-angular attention module；2.提出能有效减小冗余的encoder-decoder network）

IV.Spatial-Angular Attention Network

A.Network Architecture

• 3D卷积层
• encoder
• decoder

B.Spatial-Angular Attention Module

V.Network Training

A.Spatial-Angular Perceptual Loss

• Spatial-Angular Peceptual Loss采用的是一个3D encoder-decoder network（见上图）来提取高维特征
• 3D encoder-decoder network中编码端一步一步降维，解码端一步一步升维
• 总Loss由高维特征loss（$L_{feat}$）和像素级loss（$L_{pix}$）组成
  
  其中
  

B.Training Data

• data augmentation
  采用shearing操作来做数据增强：
  
  

C.Implementation Details

VI.Evaluations

• 在Gantry Systems和Lytro Illum两个数据集下做评估；做了消融实验（without the SAAM,without the multi-scale renconstruction structure,without the spatial-angular perceptual loss）

A.Evaluations on Light Fields from Gantry Systems

• results on the CIVIT Dataset
  

• results on the MPI Light Filed Archive
  

• Ablation studies
  具体见Table III

B.Evaluations on Light Fields from Lytro Illum

• error map
  
• quantitative results
  

C.Further Analysis

• Spatial-angular attention map
  
• Computational time
• Limitations

VII.Conclusions

个人总结

参考

[1] 吴高昌个人主页

码字不易，如果您觉得有帮助，欢迎点赞和收藏~~