A Measurement-Based Skin Reflectance Model for Face Rendering and Editing

引言：https://www.cnblogs.com/sdnfever/p/3753935.html

“BSSRDF是Bidirectional Scattering Surface Reflectance Distributed Function双向散射表面反射率分布函数的缩写。说到BSSRDF就不得不先说BRDF。BRDF是Bidirectional Reflectance Distributed Function双向反射率分布函数的缩写。

要完备地描述真实世界中的材料的反射性质，就必需描述在各个方向入射到材料上的光线在各个方向的反射强度按反射方向的分布情形。这就是所谓的“双向反射率分 布”。其中完全镜面折/反射光强按折/反射角的分布情形是由物理学家A.J.Fresnel（菲涅耳）总结出来的，被称为菲涅耳公式。而计算机图形学中长 久以来使用的光照模型（如，phong，lambert，blinn等）就是物理学范畴中的BRDF的近似。

然而，即使是物理学范畴中的BRDF都有不完备之处------它是在假定光线的入射和反射发生在材料表面同一点的前提下进行统计的。而光在射入真实世界中 的绝大多数材料表面时都会扩散到材料内部，然后再经过各不相同的路径反射出来，总而言之不是限定在材料表面同一点发生。这样一来恐怕只有金属才是严格的 BRDF，其它能发生或强或弱散射的材料都可归为BSSRDF。云，大理石，葡萄，牛奶，人体的各种生物组织都是明显表现半透明的材料，都是因高度散射造 成的。要表现这些特性用BRDF显然力不从心。而BSSRDF却能兼容所有材料的属性，包括BRDF。可见不能忽视BSSRDF。

Abstract
We present a novel skin reflectance model for faces and its application to face appearance editing. we decompose the high-dimensional bidirectional scattering surface reflectance distribution function (BSSRDF) of skin into components that can be estimated from measured data. our model is intuitive, amendable to interactive rendering, and easy to eidt. high-quality renderings come close to reproducing real photographs. we have measured 3D face geometry, skin reflectance,and substance scattering for a large group of people using custom-built devices and fit the data to our model. the analysis of the reflectance data reveals variations according to subject age, race, gender and external factors (heat, cold, makeup, etc.) we derive a low-dimensional model using non-negative matrix factorization (NMF) that spans the space of skin reflectance in our database. a user can define meaningful parameters in this space-such as race, gender, and age -and change the overall appearance of a person (e.g., making a Caucasian face look more Asian) or change local features (e.g., adding models, freckles 雀斑 or hair follicles 毛囊).

1 Introduction
one of the most difficult computer graphics challenges is creating realistic human faces. humans have evolved to be incredibly adept at interpreting facial appearance 识别面部表情。 for example, we can easily distinguish if a person is tired, hot, excited, or sick. although a lot of effort has been devoted to face modeling in computer graphics, no synthetic face model to data achieves this level of expressiveness and realism. 没有任何一个人脸合成模型能达到这种表达能力和真实性。

in this paper, we focus on modeling skin refletance of human faces, an important aspect of face appearance. it varies for different people (e.g., due to race or gender) and even varies for the same person throughout the course of a day (e.g, ht vs. cold skin). a realisitc skin reflectance model should be able to accomodate 适应 these variations. it should also allow a graphic artist to change the appearance of skin based on easy to interpret parameters (e.g., race, gender, or age). the model needs to easily connect to measurements of real faces for the creation of virtual doubles. images generated from the model——ideally in real-time——need to look photorealisic from arbitrary viewpoints. and the model should allow easy modification or transfer of skin appearance.

To achieve these goals we have developed a novel skin reflectance model whose components can be robustly estimated from measured data. Our model is accurate, compact, and intuitive to edit. It can be used in interactive and offline rendering systems and generates results that come close to reproducing real photographs. We use custom-built devices to measure in-vivo light reflection and subsurface scattering of a large and diverse group of people. 1 Our data ranges across age (13 to 74 years old), gender, race, and external factors (e.g., cosmetics, cold, and sweat). We fit our model to the measured data and compute a low-dimensional face reflectance space using non-negative matrix factorization (NMF) [Lee and Seung 1999]. User-defined parameters – such as gender, race, or tan – allow us guide the interpolation of reflectance data to change the appearance of a face overall or locally.

2 PreviousWork
Properties of human skin have been measured and studied extensively in the biomedical 生物医学, cosmetics 化妆品, and computer vision communities. In this section we provide an overview of the relevant work in the area of computer graphics and image synthesis.

Analytic Skin Reflectance Models:
Analytic reflectance models are attractive because of their computational efficiency. Hanrahan and Krueger [1993] modeled single-scattering of light in skin composed of multiple smoothly-bounded internal layers. Ng and Li [2001] extended this model by adding an oil layer to the skin surface. Stam [2001] developed an analytic approximation to multiple subsurface scattering in skin with a rough surface.
More recent work [Krishnaswamy and Baranoski 2004] proposes a biophysically-based 生物物理 multi-layer model for image synthesis with biologically meaningful parameters.
Several skin modeling approaches use analytic bidirectional surface reflectance functions (BRDFs) [Blanz and Vetter 1999; Debevec
et al. 2000; Haro et al. 2001; Paris et al. 2003; Tsumura et al. 2003; Fuchs et al. 2005]. The BRDF parameters can be estimated
from reflectance measurements using non-linear optimization. Although a BRDF describes local light transport at each surface point,
it ignores subsurface scattering, which is largely responsible for the appearance of skin.

Jensen et al. [2001; 2002] propose an analytic model for the bidirectional surface-scattering distribution function (BSSRDF). The
BSSRDF describes the full effect that incident light at a point has on the reflected light from a surface patch around that point. The
BSSRDF is eight-dimensional, assuming a two-dimensional parameterization of the surface. Because dense sampling of an eight dimensional function is challenging, we subdivide the BSSRDF into components that can be more easily measured (see Section 3).

Non-parametric Skin Reflectance Models: Instead of fitting an analytic BRDF model, Marschner et al. [1999] estimate a nonparametric
BRDF of a human face by combining reflectance samples from different points on the surface. They later extended this work by adding a detailed albedo texture [Marschner et al. 2000]. They observe that the BRDF of skin is quite unusual and exhibits strong forward scattering at grazing angles that is uncorrelated with the specular direction. We use the data-driven BRDF model of Matusik et al. [Matusik et al. 2003] to estimate a non-parametric surface BRDF at each surface point. We found that this introduces less error than imposing the behavior of a particular analytic BRDF model. More importantly, it does not require non-linear optimization and leads to a
more robust fitting procedure.

Image-based Face Modeling: Image-based methods have provided highly realistic representations for human faces. They easily
capture effects such as self-shadowing, inter-reflections, and subsurface scattering [Pighin et al. 1998]. Recent efforts allow variations
in lighting [Georghiades et al. 1999; Debevec et al. 2000], viewpoint, and expression [Hawkins et al. 2004]. Cula et al. [2005;
2004] collected a database containing more than 3500 skin texture images that were taken under various illumination and viewing conditions [Rutgers ]. However, the memory requirements for imagebased models are large. The measurement procedures are inefficient and assume non-local low-frequency lighting. Pure image-based representations are also inherently difficult to edit and modify. Borshukov and Lewis [2003] combine an image-based model, an analytic surface BRDF, and an approximation of subsurface scattering to create highly realistic face images for the movie Matrix Reloaded. Sander et al. [2004] developed a variant of this method
for real-time skin rendering on modern graphics hardware.
An interesting image-based method was presented by Tsumura et al. [2003], who use independent component analysis (ICA) to
decompose images of faces into layers (melanin and hemoglobin). Their method is capable of re-synthesizing new images while adding effects like tanning 晒黑 or aging.

3 skin reflectance model
overal skin reflectance can be described as the sum of specular reflection of the skin surface (air-oil interface) and diffuse reflection due to subsurface scattering (see figure 2). diffuse subsurface

figure 2: skin reflectance can be explained by a specular (BRDF) component at the air-oil interface, and a diffuse reflectance component due to subsurface scattering. most of the high-frequency spatial color variation in human skin is due to the epidermal layer, whereas strong light scattering is the dermal layer is a more slowly varying effect. we model the first (high-frequency) effect with an albedo map and the second (low-frequency) light transport with a translucency map.

scattering is due to absorption and light scattering in the epidermal and dermal skin layers. The epidermis scatters light strongly and contains melanin 黑色素 (along the interface to the dermis layer), which is highly absorbing. this absorption component is a local effec with high spatial variation across the face due to hair folliles 毛囊, sweat glands 汗腺, freckles 雀斑, dimples 酒窝, etc. the dermis/blood layers is highly scattering in the red channel and strongly absorbing in the greeen and blue channels (mainly due to haemoglobin血红蛋白). the dermal light scattering is a non-local, slowly varying effect.

we model the light that is immediately reflected from the oil-skin layer with a spatially-varying surface BRDF and divide diffuse subsurface reflectance into two components: A diffuse albedo map that captures high-frequency color variations due to epidermal absorption and scatterng, and a translucency map that captures low-frequency absorption and scattering in the dermal layer. fine-scale face geometry is represented by a normal map.
More formally, we denote the BSSRDF as S(xi,ωi, xo,ωo), where ωi is the direction of the incident illumination at point xi, and ωo is the observation direction of radiance emitted at point xo. Similarly, we use fs(xi,ωi,ωo) for the surface BRDF. The relative contributions of the surface BRDF and the diffuse reflectance due to subsurface scattering are modulated by Fresnel coefficients:

where Fr and Ft = (1 - Fr) are the Fresnel coefficient at the air-skin boundary for both the incoming and outgoing radiance, and η is the
relative index of refraction between skin and air (约等于 1.3). We model the subsurface scattering term S using the dipole diffusion approximation [Jensen et al. 2001], while the specular BRDF component fs is modeled using a data-driven approach. The parameters of both the BSSRDF and the BRDF are estimated from measurements as described in the following sections.

4 Measurement Procedure Overview 测量过程概述
A block diagram of our measurement pipeline is shown in Figure 3. We capture the 3D geometry of the face using a commercial 3D
face scanner. Digital photographs from different viewpoints and

Figure 3: A block diagram of our data processing pipeline. Blocks in grey are the parameters of our skin reflectance model.

with different illumination directions are taken in a calibrated facescan dome. The data is used to compute a normal map and to estimate the diffuse reflectance at each surface point. We subtract the diffuse reflectance from the measured data and fit a set of denselymeasured BRDFs to the remaining surface reflectance. We compress the BRDF basis using NMF to derive a small set of NMF basis BRDFs. We then measure the subsurface scattering of skin at few locations in the face using a special contact device and estimate skin translucency.
To map between 3D face space and texture space we use the area-preserving texture parameterization of Desbrun et al. [2002]. The data is densely interpolated using push-pull interpolation into texture maps of 2048×2048 resolution. The parameters of our reflectance model are the NMF basis BRDFs (typically four), textures with coefficients for the linear combination of basis BRDFs, one albedo map with diffuse reflectance values, and one translucency map. The following sections describe each of these processing steps in more detail.

5 measuring skin reflectance
figure 4 shows a photograph of our face-scanning dome 面部扫描圆顶. the subject sits in a chair with a head rest to keep the head still 保持头部不动 during the capture process. 拍摄过程中，受试者坐在椅子上，头靠在椅子上以保持头部不动。

the chair is surrounded by 16 cameras and 150 LED light sources that mounted on a geodesic dome. the system sequentially turns each light on while simultaneously capturing images with all 16 cameras. We capture high-dynamic range (HDR) images [Debevec and Malik 1997] by immediately repeating the capture sequence with two different exposure settings. The complete sequence takes about 25 seconds 花费25秒 for the two passes through all 150 light sources (limited by the frame rate of the cameras). To minimize the risk of light-induced seizures 光诱发癫痫 we ask all subjects to close their eyes. We report more details about the system and its calibration procedure in [Anonymous 2005].(2)
(2) this technical report has been submitted as supplement material.本技术报告已作为补充材料提交

a commercial face-scanning system from 3QTech (www.3dmd.com) is placed behind openings of the dome. using two structured light projectors and four cameras, it captures the complete 3D face geometry in less than one second. the output mesh contains about 40,000 triangles and resolves features as small as 1 mm. we clean 清除 the output mesh by mannually cropping 手工清除 non-facial areas and fixing non-manifold issues and degenerate triangles. the cleaned mesh is refined using loop-subdivision [loop 1987] to obtain a high-resolution mesh with 500,000 to 1 million vertices. the subdivision implicitly removes noise. we store the high-resolution mesh as an unstructured list of point samples (surfels) 不知道什么意思 without connectivity. each surfel stores the necessary in formation for image reconstruction using EWA splatting [Zwicker et. al. 2002].

next we compute a lumitexel [Lensch et al.2001] at each surfel position from the image reflectance samples. each observed radiance value L(ωo) is normalized by the irradiance, Ei(ωi), of the corresponding light source l in order to obtain a BRDF sample value:

We calibrated 校准 the BRDF measurements using Fluorilon — a material with know properties我们使用一种已知性能的材料氟纶校准了BRDF的测量值 [Anonymous 2005]. All processing is performed on RGB data except 除非 where noted otherwise. 所有处理都是对RGB数据执行的，除非另有说明。

we determine lumitexels in shadow areas by rendering the face from camera and light source viewpoints using EWA splating [Zwicker et al. 2001]. EWA filtering guarantees that each surfel splat covers at least one pixel. the percentage of visible pixels per splat is approximated by the ratio of total rasterized pixels to effectively blended pixels. if this percentage is above a threshold we mark the surfel as visible from that carmera or light source. this process of determing visibility and occlusion is similar to shadow mapping [williams 1978], but using EWA splatting. 我理解的splat是斑点的，色块的意思

figure 5 shows a visualization of lumitexels from two different points in the same face. the images given an impresion of the hemispherical sampling for each camera viewpoint. on average, a lumitexel contains about 900 reflectance samplers per color channel, with many lumitexels having up to 1,200 samples.

figure 5: visualization of two skin lumitexels for different viewpoints. only the samples for a single camera are shown. the red and green dots are intersection of the view vector and reflection vector with the hemisphere, respectively. reflectance values for each light position (blue dots) are used to shade the hemispherical triangulation. note that the reflection direction corresponds well with the speuclar highlight on the left. the right lumitexel is more diffuse, and some light positions are occluded by the nose 鼻子.

the numbers vary depending on the occlusions across the face (see figure 5). in constrast to previous work [Maraschner et al. 1999; Lensch et al. 2001; Fuchs et al. 2005], We collect enough samples for a reliable BRDF fit at almost all lumitexels without clustering. 我们收集了足够的样本，几乎在所有的lumitexels上都能得到可靠的BRDF拟合，而不需要聚类。the data for lumitexels with a badly conditioned fit is interpolated (插值，with……修饰的是lumitexels) during creation of the texture maps. 在创建纹理贴图的过程中，对拟合条件很差的lumitexels的数据进行插值被插值。

6 Estimating normals and diffuse albedo
We estimate normals at each lumitexle from the reflectance data. for each camera viewpoint, we determine the direction of maximum reflectance by interpolation between the directions of the four brightest measurements. the half-way vector between this direction and the viewing vector is the normal estimate for this viewpoint. we then remove outliers from the set of normals from all viewpoints and interpolate the final surfel normal. this normal estimation procedure is stable and leads to good results considering the precision of our measurements (see Figure 6). however, real-world reflections are off-sepcular, i.e., they are not necessarily aligned with the mirror direciton. to imporve the normal estimation further we could use photometric stereo methods [Goldman et al. 2003].

To separate specular surface reflectance from diffuse subsurface reflectance at each surfel we use the diffuse BRDF approximation of the BSSDF by Jensen et al.[2001]:

where Rd is the diffuse reflectance:

Rd depends only on the reduced albedo α’ and the Fresnel terms. The BRDF approximation is equivalent to the full BSSRDF for a semi-infinite plane of homogeneous, almost opaque material under uniform incident illumination. Intuitively, under these assumptions we cannot distinguish if photons enter the surface, scatter, and reemit as diffuse light, or if they are immediately reflected at a point from a diffuse surface. We found this approximation to be relatively accurate for the diffuse reflectance component of our skin model. Any remaining error will show up as a diffuse component in the surface BRDF.

We estimate Rd at each surface point from the lumitexel data. Based on the assumption that we observe pure diffuse reflectance for at least some of the observation angles, We are looking for the maximum value Rd that is less than or equal to the minimum of the observed BRDF values fr. This is done by determining the minimum ratio between fr and the unit diffusion reflectance:

Note that we divide the observed BRDF by the Fresnel coefficients in accordance with Equation (2) as we compute the diffuse component using the BSSRDF. In order to reduce outliers and the influence of motion artifacts, we determine a stable minimum by penalizing grazing observations and discarding the k smallest values. The Rd values for each surface point are re-parameterized and interpolated into the albedo map.
注意，当我们使用BSSRDF计算漫反射分量时，我们根据公式(2)将观察到的BRDF除以菲涅尔系数。为了减少异常值和运动伪影的影响，我们通过惩罚擦边观测值和丢弃k个最小值来确定一个稳定的最小值。每个表面点的Rd值被重新参数化并插入到反照率地图中。