文献阅读(二)

文献阅读

Learning Efficient Illumination Multiplexing for Joint Capture of Reflectance and Shape

一. 问题背景

1.问题介绍

• joint acquisition of unknown reflectance and shape
• efficient capture of both reflectance and shape is fundamentally challenging
  (1)渲染方程的高维性以及与image-based measurement的紧密联系.所以需要 take as many measurements as possible来得到足够的信息
  James T. Kajiya. 1986. The Rendering Equation (SIGGRAPH ’86). 143–150.
  (2)实际应用中,the number of samples can be
  strictly limited.而其解决方法是优化acquisition efficiency

2.相关工作

2.1 综述

样品反射率和形状的重建可以分为两个种类,分类是基于入射光是否被控制.在相关工作中仅介绍被控制的情形.可参考下面的综述.

• Yue Dong. 2019. Deep appearance modeling: A survey. Visual Informatics (2019).
• Darya Guarnera, Giuseppe C. Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF Representation and Acquisition. Computer Graphics Forum 35, 2 (2016), 625–650
• Michael Weinmann and Reinhard Klein. 2015. Advances in Geometry and Reflectance Acquisition. In SIGGRAPH Asia Courses. Article 1, 71 pages.
• Tim Weyrich, Jason Lawrence, Hendrik P. A. Lensch, Szymon Rusinkiewicz, and Todd Zickler. 2009. Principles of Appearance Acquisition and Representation. Found.Trends. Comput. Graph. Vis. 4, 2 (2009), 75–191.

2.2 Geometry Reconstruction with the Diffuse Assumption

• 下面的方法是基于diffuse-dominant reflectance假设(它是多个view 下的不变量,用来在multi-view之间建立对应),但是对于被general SVBRDF表达的外观就不行了:" the reflectance that changes with the view is often treated as outliers, or physically modified via means like powder coating."
  (1)Daniel Scharstein and Richard Szeliski. 2003. High-accuracy stereo depth maps using structured light. In CVPR.
  高质量的几何重建可以基于active illumination方法,比如strutured lighting
  (2)Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In ECCV.
  对于丰富表面纹理的样品,passive的方法,比如structure-from-motion也可以很好的恢复形状
• Robert J Woodham. 1980. Photometric method for determining surface orientation from multiple images. Optical engineering 19, 1 (1980), 191139.
  另一条线是 photometric stereo.从不同光照条件下外观的变化,它估计法向场,而法向场可以合并到一个3D的表面上去.
  这一种方法,最近的研究仍然局限于各向同性的镜面反射光,并需要 96 distant lighting conditions
  Satoshi Ikehata. 2018. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In ECCV

2.3 Spatially-Varying Reflectance Capture on a Known Shape

• 对6D定义域的SVBRDF的直接采样,通过mechanically positioning a camera and a light source是及其费时的
  Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. 1999.
  Reflectance and Texture of Real-world Surfaces. ACM Trans. Graph. 18, 1 (Jan. 1999),
  1–34.
  Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter
  Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse Shade Trees
  for Non-parametric Material Representation and Editing. ACM Trans. Graph. 25, 3
  (July 2006), 735–745.
  对于反射率数据的priors被引入减少消耗,包括基本材料的线性组合:
  Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter
  Seidel. 2003. Image-based Reconstruction of Spatial Appearance and Geometric
  Detail. ACM Trans. Graph. 22, 2 (April 2003), 234–257
  Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous Localization and
  Appearance Estimation with a Consumer RGB-D Camera. IEEE TVCG 22, 8 (Aug
  2016), 2012–2023.
  以及反射率位于低维流形上:
  Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and
  Baining Guo. 2010. Manifold Bootstrapping for SVBRDF Capture. ACM Trans.
  Graph. 29, 4, Article 98 (July 2010), 10 pages.
  以及stochastic-texture-like的材料的假设
  Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF Capture for
  Stationary Materials. ACM Trans. Graph. 34, 4, Article 110 (July 2015), 13 pages.
• Illumination-multiplexing-based approaches可以有效的获取高质量的结果,大量的光宇被同时program
  (1)Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. 2009.
  Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumination. Computer Graphics Forum 28, 4 (2009), 1161–1170.
  通过捕捉spherical harmonics lighting patterns下的照片,然后通过人工推导的反向查询表(which maps the observed radiance to BRDF parameters)来恢复反射率
  (2)Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance
  Scanning: Estimating Shading Frame and BRDF with Generalized Linear Light
  Sources. ACM Trans. Graph. 33, 4, Article 117 (July 2014), 11 pages.
  Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. 2003. Linear light
  source reflectometry. ACM Trans. Graph. 22, 3 (2003), 749–758.
  线性光源在平面材料的样品上移动,并通过对应的外观变量对SVBRDF进行重建
  (3)Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF Capture in
  the Frequency Domain. ACM Trans. Graph. 32, 4, Article 110 (July 2013), 12 pages.
  使用相机和近场的LCD板作为光源,基于频域分析,来获得各向同性的反射率
• Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient
  Reflectance Capture Using an Autoencoder. ACM Trans. Graph. 37, 4, Article 127
  (July 2018), 10 pages.
  就是上一篇阅读的文章.那一篇文章不能直接向这里推广的原因是 " the extra complexity of unknown geometry, and the complicated interplay between reflectance and shape in image measurements."

2.4 Joint Acquisition of Reflectance and Shape

• Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet
  Ghosh, and Paul Debevec. 2013. Acquiring Reflectance and Shape from Continuous
  Spherical Harmonic Illumination. ACM Trans. Graph. 32, 4, Article 109 (July 2013),
  12 pages.
  rotating LED arc去向样品上投射连续的SH pattern.它是基于distant lighting assumption的.per-pixel reflectance maps给每个view计算,然后将其作为shape reconstruction的输入.这个方法不能用于该方法的原因是:为了减少反射率重建的近场效应,需要每个像素精确的3D位置.
• Zhenglong Zhou, Zhe Wu, and Ping Tan. 2013. Multi-view photometric stereo with
  spatially varying isotropic materials. In CVPR.
  这篇文章是基于 isotropic reflectance assumption的,不能用于aisotropic的情形.利用多个view ,环状LED灯每次只开一个.先进行isotropic reflectance computation,然后去estimate the geometry.稀疏的灯可以防止per-pixel reflectance estimation
• Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A Coaxial Optical Scanner
  for Synchronous Acquisition of 3D Geometry and Surface Reflectance. ACM Trans.
  Graph. 29, 4, Article 99 (July 2010), 12 pages.
  通过投影仪-相机对以及phase-shift pattern,使用建立了一个gantry,来进行几何重建.但是需要对恢复的反射率比较强的prior,这是由sparse sampling in the angular domain导致的假设.
• Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering Shape and Spatiallyvarying Surface Reflectance Under Unknown Illumination. ACM Trans. Graph. 35,
  6, Article 187 (Nov. 2016), 12 pages
  利用未知光照的不连续性,从旋转物体的video sequence来恢复形状和各向同性的反射率.
  而
  Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical SVBRDF
  acquisition of 3D objects with unstructured flash photography. In SIGGRAPH Asia
  Technical Papers. 267
  " take hundreds of flash photographs from multiple views, to compute a 3D geometry and isotropic reflectance expressed as a linear combination of basis materials, via an involved alternating optimization. "

2.5 Deep-Learning-Assisted Modeling

• 对于将Deep neural network使用到反射率重建中,可以参考下面的文章
  (1)Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien
  Bousseau. 2018. Single-image SVBRDF Capture with a Rendering-aware Deep
  Network. ACM Trans. Graph. 37, 4, Article 128 (July 2018), 15 pages.
  (2)Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling Surface Appearance
  from a Single Photograph Using Self-augmented Convolutional Neural Networks.
  ACM Trans. Graph. 36, 4, Article 45 (July 2017), 11 pages.
• 对于将Deep neural network使用到形状重建中,可以参考下面的文章
  (1)Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy,
  Abraham Bachrach, and Adam Bry. 2017. End-To-End Learning of Geometry and
  Context for Deep Stereo Regression. In ICCV.
  (2)Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018. MVSNet: Depth
  Inference for Unstructured Multi-view Stereo. In ECCV.
• 最近

Shihao Wu, Hui Huang, Tiziano Portenier, Matan Sela, Daniel Cohen-Or, Ron Kimmel, and Matthias Zwicker. 2018. Specular-to-Diffuse Translation for Multi-View
Reconstruction. In ECCV.
…

3. ACQUISITION SETUP

见原文

4.Preliminareis

在不失一般性的前提下,这篇文章对于上面的acquisition
setup做了independently controlled, near-field or distant light sources的假设.并且假设物品是不透明的,可以使用3D mesh进行建模,并且表面形状进行各向异性的SVBRDF建模.偏振过滤器不被使用.同时每个点反射率是单独构建的,没有空间连续性的假设

关于下面几个公式的解释见上一篇文献阅读文章
B ( I , p ) = ∫ 1 ∥ x 1 − x p ∥ 2 I ( l ) Ψ ( x 1 , − ω i ) f r ( ω i ′ ; ω o ′ , p ) ( ω i ⋅ n p ) ( − ω i ⋅ n 1 ) d x 1 L 0 ( p , ω 0 ) = L e ( p , ω 0 ) + ∫ ξ 2 f r ( p , w i → w 0 ) L i ( p , ω i ) cos ⁡ θ d ω i B ( I , p ) = ∑ l I ( l ) m ( l ; p ) m ( j ; p ) = B ( { I ( l = j ) = 1 , I ( l ≠ j ) = 0 } , p ) \begin{aligned} B(I, \mathbf{p})=& \int \frac{1}{\left\|\mathbf{x}_{1}-\mathrm{x}_{\mathrm{p}}\right\|^{2}} I(l) \Psi\left(\mathrm{x}_{1},-\omega_{\mathrm{i}}\right) f_{r}\left(\omega_{\mathrm{i}}^{\prime} ; \omega_{\mathbf{o}}^{\prime}, \mathbf{p}\right) \\ &\left(\omega_{\mathbf{i}} \cdot \mathbf{n}_{\mathbf{p}}\right)\left(-\omega_{\mathbf{i}} \cdot \mathbf{n}_{1}\right) d \mathbf{x}_{1} \end{aligned}\\ L_{0}\left(p, \omega_{0}\right)=L_{e}\left(p, \omega_{0}\right)+\int_{\xi^{2}} f_{r}\left(p, w_{i} \rightarrow w_{0}\right) L_{i}\left(p, \omega_{i}\right) \cos \theta d \omega_{i}\\ B(I, \mathbf{p})=\sum_{l} I(l) m(l ; \mathbf{p})\\ m(j ; \mathbf{p})=B(\{I(l=j)=1, I(l \neq j)=0\}, \mathbf{p}) B(I,p)=​∫∥x1​−xp​∥21​I(l)Ψ(x1​,−ωi​)fr​(ωi′​;ωo′​,p)(ωi​⋅np​)(−ωi​⋅n1​)dx1​​L0​(p,ω0​)=Le​(p,ω0​)+∫ξ2​fr​(p,wi​→w0​)Li​(p,ωi​)cosθdωi​B(I,p)=l∑​I(l)m(l;p)m(j;p)=B({I(l=j)=1,I(l​=j)=0},p)
更进一步的将,lumitexel m可以表示成diffuse lumitexel $m_d$和specular lumitexel $m_s$的和.$$m(l)=m_d(l)+m_s(l)$$(个人理解,这里应该是将渲染方程中的$f_r$给拆了)

5.Overview

本文通过小样本的,同样的lighting pattern下的multi-view photographs,利用mixed-domain neural network取捕捉形状以及反射率.对于每个view中的有效的pixel location,通过投射不同的光照模式,network 将对应点p上面的lumitexel 给encode成少量的measure values.然后将measurements给decode 成diffuse/specular lumitexels,法向向量以及估计的位置.根据不同的view下的这些信息,可以计算出在多角度下的详细的3D mesh.当shape确定了以后,在每一个表面的点上可以将一个4D BRDF以及局部坐标系与lumitexel匹配,然后产生代表了最后的6D SVBRDF的texture map

5.1 Design Considerations

• 不会直接使用end-toend(?)为最终的几何学习估计的位置p.原因是因为虽然lumitexel中包含了与位置相关的信息(即$\frac{-{\omega }_i\cdot n_l}{|||x_1-x_p|{|}^2}$),但是这个信息对p的变化太敏感了,很难得到高精度的3D位置信息,但是这个p又需要足够的准确用来在decode diffuse lumitexel for estimating ${\rho }_d$时eliminate near-field effect.
• 不像之前的文章,是利用得到的reflectance来得到normal,而是直接得到这个信息.这是因为在近场条件下,反射率和几何是highly coupled(" the reflectance reconstruction requires a position at the current pixel, while the shape reconstruction takes normals as input ").本文中利用深度网络来拆解这个mutual dependency.
• 与上一篇阅读的文章类似,选择学习lumitexs,而不是直接得到BRDF参数(" due to the simple spatially invariant, linear relationship among the lighting pattern, the lumitexel and the measurements ").
• 现在的主要工作是将multi-view stereo的信息聚合起来建立多个view下的reliable的对应,这是几何重建下重要的异步.但是这个netwok将每一个点的lumitexel当做输入,原因是这使得network简单,并且阻值了**"** the possible combinatorial explosion in synthesizing training data of varied reflectance and shape ".并且利用state-of-the-art existing work for spatial aggregation 而不是加重网络的负担.

6.NETWORK