Spatial-Angular Attention Network for Light Field Reconstruction【论文阅读笔记】
作者
Gaochang Wu, Yebin Liu, Lu Fang, Tianyou Chai
相关链接
- Paper:https://arxiv.org/pdf/2007.02252.pdf
- Code:https://github.com/GaochangWu/SAAN
- Video:https://www.bilibili.com/video/av498739331/
需要补的知识点/论文
- non-local attention machanism
- U-net
Abstract
I.Introduction
- non-local attention machanism?
- multi-scale reconstruction structure?
- U-net?
an encoder-decoder structure with skip connections - 3D auto-encoder
3 contributions
- A spatial-angular attention module(SAAM) that perceives correspondences non-locally in the epipolar plane;(non-local attention machanism)
- A multi-scale reconstruction structure for efficiently performing the non-locall attention in the low spatial scale while also preserving the high frequencies;(multi-scale reconstruction structure;U-net)
- A spatial-angular perceptual loss specifically designed for the high-dimensional light filed data.(through 3D auto-encoder)
II.Related Work
A.Light Filed Resconstruction
- Depth image-based view synthesis
- Reconstruction without explicit depth
B.Attention Machanism
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS, 2017, pp. 5998–6008.
- X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7794–7803.
- H. Zhang, I. Goodfellow, D. N. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” arXiv: Machine Learning, 2018.
III.Problem Analysis
-
EPI像素的斜率越陡,视差越小
-
大的感受野会有更好的重建效果
-
但受参数量限制,没法一直加大卷积核大小,作者提出使用non-local perception(1.受non-local attention mechanism的启发,提出spatial-angular attention module;2.提出能有效减小冗余的encoder-decoder network)
IV.Spatial-Angular Attention Network
A.Network Architecture
- 3D卷积层
- encoder
- decoder
B.Spatial-Angular Attention Module
V.Network Training
A.Spatial-Angular Perceptual Loss
- Spatial-Angular Peceptual Loss采用的是一个3D encoder-decoder network(见上图)来提取高维特征
- 3D encoder-decoder network中编码端一步一步降维,解码端一步一步升维
- 总Loss由高维特征loss(
L
f
e
a
t
L_{feat}
Lfeat)和像素级loss(
L
p
i
x
L_{pix}
Lpix)组成
其中
B.Training Data
- data augmentation
采用shearing操作来做数据增强:
C.Implementation Details
VI.Evaluations
- 在Gantry Systems和Lytro Illum两个数据集下做评估;做了消融实验(without the SAAM,without the multi-scale renconstruction structure,without the spatial-angular perceptual loss)
A.Evaluations on Light Fields from Gantry Systems
-
results on the CIVIT Dataset
-
results on the MPI Light Filed Archive
-
Ablation studies
具体见Table III
B.Evaluations on Light Fields from Lytro Illum
- error map
- quantitative results
C.Further Analysis
- Spatial-angular attention map
- Computational time
- Limitations
VII.Conclusions
个人总结
参考
码字不易,如果您觉得有帮助,欢迎点赞和收藏~~