Towards Real-Time Photorealistic Rendering: Challenges and Solutions

https://www.researchgate.net/publication/228890410_Antialiasing_of_Bump_Maps 清晰版

Abstract
A growing number of real-time applications need graphics with photorealistic quality, especially in the field of training (virtual operation, driving and flightsimulation), but also in the areas of design or ergonomic research. We take a closer look at main deficiencies of today’s real time graphics hardware and present solutions for several of the identified problems in the areas of antialiasing and texture-, bump- and reflection mapping. In the second part of the paper, a new method for antialiasing bump maps is explained in more detail. CR Categories and Subject Desscriptors: I.3.1 [Computer Graphics]: Hardware Architecture – Graphics Processors, Raster 光栅 Display Devices; I.3.3 [Computer Graphics]: Picture/Image generation.

Additional Key Words and Phrases: antialiasing, bump mapping, environment mapping, anisotropic filtering.

1. Introduction
Using elaborate 精细制作的 software techniques, it is possible to render images that resemble photographs. These techniques are commonly applied in the movie industry, but unfortunately, they are not feasible for hardware implementation. A recently rendered movie took a week of rendering for 3.5 minutes of the movie on the PIXAR RenderFarm. Nevertheless, high quality is needed in several application areas as, e.g., ergonomics research 人体工程学研究, accident research or training 事故研究或培训. But even expensive high-end graphics hardware suffers from some important deficiencies 缺陷 . These deficiencies provide the graphics hardware researcher with interesting challenges for the next few years. The following sections identify some of the challenges for computer graphics hardware and point towards solutions.

Complexity
With today’s technology, it is easy to create scenes with hundreds of millions of triangles. Using 3D-scanning, even more complex
scenes can be captured. One answer to the problem of complexity is, of course, speed.
Replacing off-chip data-transfer by on-chip-transfer will provide us with speedups of orders of magnitude within the next few years [9]. But even such speedups will not suffice for the growing demands. Methods to reduce the number of triangles that have to be rendered will therefore become more and more important:
Simplification and Visibility Culling. Texture mapping and imagebased rendering are already state of the art and will be treated in an own section. Mesh simplification and multiresolution modeling [7][8]are however performed in software and are both well suited to general-purpose architectures. Together with visibility culling, these software techniques are an important part of Silicon Graphics (美国硅图公司（SGI，Silicon Graphics）) new OpenGL++ or Optimizer package. Visibility culling 可见性剔除, especially occlusion 遮蔽剔除 culling is up to now 到现在为止 also performed in software, but some of the algorithms could probably well be supported with suitable hardware.

Texture Filtering
texture mapping is a common means to improve the appearance of computer generated images. it allows to reduce the number of rendered polygons significantly, especially in combination with mesh simplification techniques. special techniques, like the imposters proposed by Schaufler et.al. [10] allow to exploit frame coherency 一致性 using texture mapping. hardware support for texture mapping can be found even on cheap graphics boards for the PC. it is, however, astonishing that from low-end- to high-end systems, the hardware supports only isotropic filtering with mip-maps. isotropic filtering leads to very blurry images, if the viewing angle is small. we have proposed footprint-assembly, as a simple method to perform anisotropic filtering that requries only minimal modifications to the mip-mapping hardware [11]. for a comparison of the results see fig.1 and fi.2. new, completely image based rendering techniques[5] can be supported by slightly modified texture mapping hardware. further research in this area is needed.

Surface appearance
an other challenge is the appearance of surfaces. often, it is a single surface that makes it easy to tell a computer generated image from a photograph. this is the even for ray-traced images, that

3. antialiasing of bump maps
   as mentioned, bump-maps, like texture maps or any other maps consisting of discretely stored data have to be properly filtered, if they are being resampled in the process of rendering an image. if, however, bump maps are filtered in the traditional way, the bumps are lost in the filtering process and the result is a smooth surface. we introduce a bump map pyramid, that contains and preserves isotropic or anisotropic roughness information in all resolution levels.

4. problem
   the antialiasing of bump maps is similarly important as the antialiasing of textures in general (fig. 3). however, filtering bump maps in the common way does not lead to the desired result, but instead results in a smoothened surface. the lower the resolution gets, the more we have to fitler, the larger are the bumps that disappear (Fig.4). so filtering bump maps means to remove the bumps. in reality however, a bumpy surface can be told from a smooth surface, even if it is viewed from a larger distance, where the individual bumps are not discernible 明显的 anymore (fig.5). what can be seen in this case, is the bumpiness or roughness of the surface. the problem is, that filtering removes not only the individual bumps, but also the bumpiness.
   bump maps can be represented as height-or offset-maps[4]. we prefer a representation that uses a two-dimensional vector field of offset vectors to be added to the normal vector, which is described below. but regardless of how a bump map is represented, filtering or averaging always leads to the mentioned effects.
   
   Fig. 1: Antialiasing of bump maps is necessary; otherwise we get rays reflected into random directions or random Phong illumination values.
   
   Fig. 2: Filtering the bump map removes bumps and produces a smooth surface

Fig. 3: The roughness of a surface causes rays from more than only one direction to be reflected into one pixel. These rays form a reflection cone or a conical reflection body

reflection cone反射椎 or conical reflection body 锥形反射体

6. a new solution
   if we want to preserve the important surface properties, in addition to the bump map we have to introduce and use a new type of maps: the roughness map. it stores the variance of the normal directions from the area that contributes to one sample in the filtered version of the bump map. consider for example a lake, seen from some distance. even if there are waves, the surface of the water within one pixels region can be represented by only one normal vector, if standard bump mapping is used. if this pixel region is large enough, this average normal vector will always point straight up. so we will get the same reflections of the sun (or any environment) as in a perfect mirror, regardless of the presence or absence of small waves. if we store, however, the roughness of the surface, we can account for the waves and get a better approximation of the lake than a perfect mirror would be. we could use roughness as a measure for the exponent in the Blinn-Phong model[3]. a better way will be described in [12], where environment maps are used and the roughness serves as a parameter for the anisotropic antialiasing of the environment map. 环境贴图被使用，粗糙度作为参数提供给各向异性的环境贴图。

Fig. 4: Representation of bump map with offset vectors.

in this chapter, we will first show a practical representation of bump maps, then introduce an anisotropic measure for the roughness, and then show, how resolution pyramids can be calculated and practically stored. we offer a cheap, isotropic and a more expensive anistopic version for the bump-roughness pyramid. examaples will illustrate the different methods.
本章，我们首先说明bump maps的使用。然后介绍一个度量粗糙度各向异性的标准。然后介绍金字塔图如何计算和存储的。我们提供一个廉价的、更加贵的各项异性的bump-roughness 金字塔。

7. the representation of bump maps.

bump mapping, even without antialiasing or environment mapping is not commonly used in real-time systems due to its heavy demands on the computing resources. this is partly, because the traditional approach to bump mapping [4] includes the calculation of the derivatives 导数of the bump function.

bump mapping with precalcualted derivatives

a possible solution that avoids the calculation of the derivatives of the bump function is to store precalcualted derivatives [11][2]. besides saving the calculation, this has the advantage, that the scaling of bump maps is as simple as the scaling of rgb-textures. if traditional bump maps are scaled with an unkown factor, it is impossible to calcualte the derivatives any more. the most difficult problem that remains to be solved is to find an appropriate local coordinate systems for each sample point, which consists of the normal vector of the surface in this point and the two tangential directions, for which the derivatives of the bump function have to be calculated (we have the same problem with the traditional representation of the bump maps). once this coordinate system has been established, the calculation of the new normal vector is performed by adding the offset vector specified by the precalculated derivatives in the local coordinate system to the surface normal.

The local coordinate system
In theory, we are free to choose a suitable local coordinate system for the perturbation of the normals by the bump maps. of course, the data stored in the bump maps is dependent of the chosen coordinate system. unfortunately, a local coordinate system that ideally fits all needs for bump mapping does not exist. but two conditions should at least be met:
1）the directions of the axes should be a continuous function of the location.坐标轴必须是关于位置的连续函数
2）the system should allow to map a bump map on any surface of an object (with any orientation), without having to recalculate the whole bump map. (this means for example, that a bump map describing the letters of the alphabet could be used to carve 雕刻 words into a surface regardless of orientation or location of the writing)

in addition, it would be nice, if the coordinate system would be orthogonal 正交的 (which is not necessarily the case for the projection of the u and v axes on the surface; this is, by the way, a problem with standard bump mapping).

for the description of some possible solutions, we will use a notation similar to that of Blinn[4]:
the points of the original patch are given by , the partial derivatives 偏导数 are given by and Pv,
pu，是对u进行求偏导；pv是对v进行求偏导。
the surface normal is thus . the sufficiently small displacement function 位移函数 f (with partial derivatives fu and fv) defines a new surface:

p’是p点沿着n/|n|进行稍微的偏移。沿着法线进行偏移？？？嗯是这个意思。

the new normal vector in this notation is

1. the natural system (perpendicular to u-lines, v-lines, n)
   we define and . we get:
   
   上图解释：
   
   the bump map stores fu and fv. 凹凸贴图存储的是fu和fv。这就是经典的非正交的u-v坐标系统进行映射bump。
   this is classical bump mapping with possibly non-orthogonal u-v coordinate system and arbitrary 任意的 |pu| and |pv|. accordant to 与什么一致 the notion of f as displacement function 位移函数, the height of the bumps is 凹凸的高度是常量 constant and does not scale with their size in the other dimensions if the map is scaled. 并且当map被缩放，这个高度依然保持不变 if the u-v mapping is continuous and well behaved, the first requirement is also fulfilled and it is possible to interpolate su and sv linearly across triangles (with perspective correction). an example for a mapping that makes problems is a sphere, if spherical coordinates are used for u and v; at the pole 极点, one coordinate is not unique. if su and sv are interpolated, the length of su and sv should be interpolated separately from the vectors themselves. as the perspective division does not change the directions of the vectors but only the lengths, only the lengths need to be corrected; the correction of the direction is performed automatically by choosing appropriately sized vectors at the vertices. the direct interpolation of two vectors with different lengths would cause the vector to change directions faster close to the end with the shorter vector. 直接对两个长度不同的向量进行插值，会使向量在接近较短向量的末端时改变方向更快。

2. the natural system with normalized unit vectors
   as pointed out in Blinn’s original bump mapping paper[4], we could also want the bump mapping to be scale invariant. for this, the perturbation 摄动 vector has to scale the same rate as n, independent of scales in p (or su, and sv). this can lead to geometrically impossible normals, if the object is scaled with different scales in u and v direcitons, which should therefore be excluded. the best choice for the coordinate system is the same as the previous one, but with normalized lengths of su and sv; thus the interpolation of the lengths can be saved. again, perspective division is not needed, as it cancels out in the normalization process.

3. an orthonormal system with main direction m
   an interesting alternative for the local coordinate system is an orthogonal system, that is derived from the normal vector n and a main direction m. the unit vectors e1 and e2 are perpendicular to n. with the help of m they are defined such, that e1 is perpendicular to m and e2 is the plane of n and m[11]. the main direction m can be e.g. one of pu or pv (in this case interpolated as above), but it can also be a constant vector for a whole object. a good example is the mapping onto a sphere like e.g. the earth with spherical coordinates. the direction of the axis of the earth would serve as m and we would get e1 pointing always in east-west direction, and e2 in south north direciton. an important advantage of a constant main direction m is that besides the normal vector, no other vector needs to be interpolated across triangles. in addition, the coordinate system can be calcualted in hardware in the rasterizer/shader and needs not to be calcualted at all vertices by a setup process.
   the calculation of the local coordinate system n, e1, e2 from the interpolated normal vector nI and the main directio m is performed using the following formula(fig. 7):
   给定主向量m，已知n，那么可以根据下面的式子求出e1、e2。
   
   fig. 7: the construction of the local coordinate system for the bump map using a main direction m.
   这个也就是根据给定的向量m，和已知的n向量，求得一个本地的坐标系统。

4. the roughness information 粗糙度信息
   有了上面的坐标系统，下面就介绍如何描述粗糙度信息。
   we have now defined, how the bump maps can be represented with precalcualted derivatives or pertubation vectors in a suitable local coordinate system. the next question is, how to represent the roughness information, i.e. the bumps that are too small to be represented in the current bump map level, but nevertheless are important for the visual appearance of the surface. this roughness information determines ultimately the size and shape of the reflection bodies (fig.5), into which the small, pixel-wide viewing beams are reflected by the rough surface. a complete description of the roughness would consist of the complete distribution of perturbation vectors within the considered area. the only difference between the roughness information and the complete detailed bump map is, that the location, where a certain perturbation vector applies, can be omitted. but the complete distribution of perturbation vectors is still much too much information to store and to use. a common way out is to assume a certain form of the distribution, which allows to represent the distribution with only a few parameters. a one-dimensional example for such parameters are average and variance in the case of a normal distribution. the counterpart of the variance in the 2D case is the two-dimensinal covariance matrix K, which describes the distribution in the form of an ellipse 椭圆:
   on this ellipse, the probability density is constant; the distance of a point X on the ellipse to the average vector X(上一横）represents the standard deviation in the direction (X-X上横) of this point. of course, the assumption of a normal distribution is not valid in the general case, but the representation of the roughness with covariance matrices of the perturbation 扰动 vectors gives us the possibility to represent the normal vector distribution with an elliptical cone 椭圆锥面, that is characterized by three parameters (plus two for the average direction). This representation is appropriate for most practical normal vector distributions and enables us to model anisotropic reflection effects and produce realistic images of corrugated 波纹 sheet iron, brushed metal, small scrapes 刮痕 on a glossy surface, waves on a lake and so on.

The calculation of the covariance matrix 协方差
the covariance matrix is calcualted from the derivatives fu and fv in the following way:

反对角线数字相同。协方差矩阵后面再复习。
This matrix describes an ellipse, shown in Fig. 8 (the one in the middle of the figure). The ellipse
contains about 63% of the pertubation vectors, if their distribution is a two-dimensional normal distribution. if, however, the perturbation vectors (fu, fv) are equally distributed 分布 in a rectangle with edge lengths a and b, the ellipse has the main axes

椭圆的两个轴，就称当矩形的两个边。
we can account for that effect by introducing a correction factor, which we will mention. We have now the parameters of an ellipse, that describes the distribution of the perturbation vectors and by this the distribution of the normal vectors.

Fig. 6: The viewing rays contributing to a pixel form a beam. This beam is reflected by a bumpy surface into the reflected viewing beam. The shape of the reflected viewing beam is approximated using the distribution of normal vectors in the reflecting area by an elliptical cone. And yes: it could be argued that the normal vectors that end in the drawn ellipse, should be drawn starting from one origin, not from the surface location, they belong to. The same applies for the reflected beam.

This information can be used in a subsequent step to shade the pixel correctly. Depending on the used shading methods, there
exist different possibilities to use the roughness ellipse: In a ray tracer, the ellipse can be used to perform antialiasing by supersampling. A second possibility is to modify the Blinn-Phong shading so that is uses the roughness information (see Appendix A). Of course, it makes no sense to make huge efforts calculating the bump map and then spoil the quality of the result by applying a simple shading model. A method, that leads to a balanced system with high image quality, is to use the roughness information for the anisotropic antialiasing of environment maps[12].

The roughness pyramid

Fig. 7 shows, how the surface information is stored in two resolution pyramids, the bump pyramid and the roughness pyramid. The bump pyramid is a standard mip map, its lower resolution levels are calculated by downfiltering the higher resolution levels.
In the roughness pyramid, however, the lower resolution levels contain not only the roughness information of the higher resolution roughness levels, but also the roughness information coming from the next level in the bump pyramid, i.e.

the roughness information representing those bumps, that are omitted from the bump map in the current bump map level. This contribution is represented with a covariance matrix, calculated as shown above. It has to be added to the downfiltered covariance matrix from the roughness map. As a result we get the recursive rule for the calculation of the roughness covariance matrix in level l (with a 2x2-boxfilter for creating the mip-map):

This formula is equivalent to the calculation of the covariance matrix for all f u and fv values in the highest resolution level for the whole area covered by the respective texel in level l. It is, however, not convenient, to store the covariance matrices themselves in the roughness map. This has two reasons: 将协方差矩阵存储在粗糙度贴图中不太方便。

· the covariance matrix contains the squares of the standard deviations; to get uniform accuracy over the whole range of
standard deviations a value proportional to the standard deviations has to be stored.
· if the roughness information is used for the antialiasing of environment maps, not the covariance matrix K is needed,
but a matrix D with .

isotropic roughness representation
a more compact roughness representation can be achieved, if we restrict ourselves to isotropic roughness information. instead of three numbers, we only store one number: the standard deviation of the perturbation vector in the direction of the largest variance. we get this number by diagonalizing the covariance matrix

Conclusion
We have introduced roughness maps and roughness resolution pyramids as a way to properly antialias bump maps. After mapping color (texture maps) and bumps (bump maps), a new quality of surface properties can now be mapped to objects. Without roughness maps it is quite difficult to render reallooking images of a large class of objects with reasonable effort. Examples are small scratches 刮痕 on glossy surfaces or brushed surfaces 磨砂表面 but also every kind of bump mapped surface viewed from some distance. The expensive alternative to process such objects would be massive supersampling 超采样耗时.