OpenGL 4 Shading Language Cookbook chapter 5——Image Processing and Screen Space Techniques

introduction

in this chapter, we will focus on techniques that work directly with the pixels in a framebuffer.
These techniques typically involve multiple passes. An initial pass produces the pixel data and
subsequent passes apply effects or further process those pixels. To implement this we make
use of the ability provided in OpenGL for rendering directly to a texture or set of textures (refer
to the Rendering to a texture recipe in Chapter 4, Using Textures).

the ability to render to a texture, combined with the power of the fragment shader, opens up a huge range of possibilities. we can implement image processing techniques such as brightness, costrast, saturation, and sharpness by applying an additional process in the fragment shader prior to output.
we can apply convolution filters such as edge detection, smoothing (blur), or sharpening.
we will take a closer look at convolution filters in the recipe on edge detection.

a related set of techniques involves rendering additional information to textures beyond the traditional color information and then, in a subsequent pass, further processing that information to produce the final rendered image. these techniques fall under the general category that is often called deferred shading.

In this chapter, we’ll look at some examples of each of the preceding techniques. We’ll start
off with examples of convolution filters for edge detection, blur, and bloom. Then we’ll move on
to the important topics of gamma correction and multisample anti-aliasing. Finally, we’ll finish
with a full example of deferred shading.
Most of the recipes in this chapter involve multiple passes. In order to apply a filter that
operates on the pixels of the final rendered image, we start by rendering the scene to an
intermediate buffer (a texture). Then, in a final pass, we render the texture to the screen
by drawing a single full-screen quad, applying the filter in the process. You’ll see several
variations on this theme in the following recipes.

Applying an edge detection filter
Edge detection is an image processing technique that identifies regions where there is a
significant change in the brightness of the image. It provides a way to detect the boundaries of
objects and changes in the topology of the surface. It has applications in the field of computer
vision, image processing, image analysis, and image pattern recognition. It can also be used
to create some visually interesting effects. For example, it can make a 3D scene look similar
to a 2D pencil sketch as shown in the following image. To create this image, a teapot, and
torus were rendered normally, and then an edge detection filter was applied in a second pass.

The edge detection filter that we’ll use here involves the use of a convolution filter, or
convolution kernel (also called a filter kernel). A convolution filter is a matrix that defines how
to transform a pixel by replacing it with the sum of the products between the values of nearby
pixels and a set of pre-determined weights. As a simple example, consider the following
convolution filter:

The 3 x 3 filter is shaded in gray superimposed over a hypothetical grid of pixels. The bold
faced numbers represent the values of the filter kernel (weights), and the non-bold faced
values are the pixel values. The values of the pixels could represent gray-scale intensity
or the value of one of the RGB components. Applying the filter to the center pixel in the
gray area involves multiplying the corresponding cells together and summing the results.
The result would be the new value for the center pixel (25). In this case, the value would be
(17 + 19 + 2 * 25 + 31 + 33) or 150.

Of course, in order to apply a convolution filter, we need access to the pixels of the original
image and a separate buffer to store the results of the filter. We’ll achieve this here by using
a two-pass algorithm. In the first pass, we’ll render the image to a texture; and then in the
second pass, we’ll apply the filter by reading from the texture and send the filtered results to
the screen.

One of the simplest, convolution-based techniques for edge detection is the so-called Sobel
operator. The Sobel operator is designed to approximate the gradient of the image intensity
at each pixel. It does so by applying two 3 x 3 filters. The results of the two are the vertical
and horizontal components of the gradient. We can then use the magnitude of the gradient
as our edge trigger. When the magnitude of the gradient is above a certain threshold, then
we assume that the pixel is on an edge.
The 3 x 3 filter kernels used by the Sobel operator are shown in the following equation:

If the result of applying Sx is sx and the result of applying Sy is sy, then an approximation of
the magnitude of the gradient is given by the following equation:

if the value of g is above a certain threshold, we consider the pixel to be an edge pixel, and we highlight it in the resulting image.

In this example, we’ll implement this filter as the second pass of a two-pass algorithm. In the
first pass, we’ll render the scene using an appropriate lighting model, but we’ll send the result
to a texture. In the second pass, we’ll render the entire texture as a screen-filling quad, and
apply the filter to the texture.

set up a framebuffer object (refer to the renderint to a texture recipe in chapter 4, using textures) that has the same dimensions as the main window. connect the first color attachment of the FBO to a texture object in texture unit zero.
during the first pass, we will render directly to this texture. make sure that the mag and min filters for this texture are set to GL_NEAREST. we do not want any interpolation for this algorithm.

provide vertex information in vertex attribute zero, normals in vertex attribute one, and texture coordinates in vertex attribute two.

the following unfirom variables need to be set from the opengl application：

Width: This is used to set the width of the screen window in pixels
ff Height: This is used to set the height of the screen window in pixels
ff EdgeThreshold: This is the minimum value of g squared required to be considered “on an edge”
ff RenderTex: This is the texture associated with the FBO

any other unfiroms associated with the shading model shoud also be set from othe opengl application.

To create a shader program that applies the Sobel edge detection filter, use the
following steps:

1. Use the following code for the vertex shader:

layout (location = 0) in vec3 VertexPosition;
layout (location = 1) in vec3 VertexNormal;
out vec3 Position;
out vec3 Normal;
uniform mat4 ModelViewMatrix;
uniform mat3 NormalMatrix;
uniform mat4 ProjectionMatrix;
uniform mat4 MVP;
void main()
{
Normal = normalize( NormalMatrix * VertexNormal);
Position = vec3( ModelViewMatrix *
vec4(VertexPosition,1.0) );
gl_Position = MVP * vec4(VertexPosition,1.0);
}

2. Use the following code for the fragment shader:

in vec3 Position;
in vec3 Normal;
// The texture containing the results of the first pass
layout( binding=0 ) uniform sampler2D RenderTex;
uniform float EdgeThreshold; // The squared threshold
// This subroutine is used for selecting the functionality
// of pass1 and pass2.
subroutine vec4 RenderPassType();
subroutine uniform RenderPassType RenderPass;
// Other uniform variables for the Phong reflection model
// can be placed here…
layout( location = 0 ) out vec4 FragColor;
const vec3 lum = vec3(0.2126, 0.7152, 0.0722);
vec3 phongModel( vec3 pos, vec3 norm )
{
// The code for the basic ADS shading model goes here…
}
// Approximates the brightness of a RGB value.
float luminance( vec3 color ) {
return dot(lum, color);
}
subroutine (RenderPassType)
vec4 pass1()
{
return vec4(phongModel( Position, Normal ),1.0);
}
subroutine( RenderPassType )
vec4 pass2()
{
ivec2 pix = ivec2(gl_FragCoord.xy);
float s00 = luminance(
texelFetchOffset(RenderTex, pix, 0,
ivec2(-1,1)).rgb);
float s10 = luminance(
texelFetchOffset(RenderTex, pix, 0,
ivec2(-1,0)).rgb);
float s20 = luminance(
texelFetchOffset(RenderTex, pix, 0,
ivec2(-1,-1)).rgb);
float s01 = luminance(
texelFetchOffset(RenderTex, pix, 0,
ivec2(0,1)).rgb);
float s21 = luminance(
texelFetchOffset(RenderTex, pix, 0,
ivec2(0,-1)).rgb);
float s02 = luminance(
texelFetchOffset(RenderTex, pix, 0,
ivec2(1,1)).rgb);
float s12 = luminance(
texelFetchOffset(RenderTex, pix, 0,
ivec2(1,0)).rgb);
float s22 = luminance(
texelFetchOffset(RenderTex, pix, 0,
ivec2(1,-1)).rgb);
float sx = s00 + 2 * s10 + s20 - (s02 + 2 * s12 + s22);
float sy = s00 + 2 * s01 + s02 - (s20 + 2 * s21 + s22);
float g = sx * sx + sy * sy;
if( g > EdgeThreshold ) return vec4(1.0);
else return vec4(0.0,0.0,0.0,1.0);
}
void main()
{
// This will call either pass1() or pass2()
FragColor = RenderPass();
}

in the render function of your opengl application, follow these steps for pass #1:

1. Select the framebuffer object (FBO), and clear the color/depth buffers.
2. Select the pass1 subroutine function (refer to the Using subroutines to select shader
   functionality recipe in Chapter 2, The Basics of GLSL Shaders).
3. Set up the model, view, and projection matrices, and draw the scene.
4. Deselect the FBO (revert to the default framebuffer), and clear the color/depth buffers.
5. Select the pass2 subroutine function.
6. Set the model, view, and projection matrices to the identity matrix.
7. Draw a single quad (or two triangles) that fills the screen (-1 to +1 in x and y), with
   texture coordinates that range from 0 to 1 in each dimension.

The first pass renders all of the scene’s geometry sending the output to a texture. We select
the subroutine function pass1, which simply computes and applies the Phong reflection
model (refer to the Implementing per-vertex ambient, diffuse, and specular (ADS) shading
recipe in Chapter 2, The Basics of GLSL Shaders).
In the second pass, we select the subroutine function pass2, and render only a single quad
that covers the entire screen. The purpose of this is to invoke the fragment shader once for
every pixel in the image. In the pass2 function, we retrieve the values of the eight neighboring
pixels of the texture containing the results from the first pass, and compute their brightness by
calling the luminance function. The horizontal and vertical Sobel filters are then applied and
the results are stored in sx and sy.
We then compute the squared value of the magnitude of the gradient (in order to avoid the
square root) and store the result in g. If the value of g is greater than EdgeThreshold, we
consider the pixel to be on an edge, and we output a white pixel. Otherwise, we output a solid
black pixel.

The Sobel operator is somewhat crude, and tends to be sensitive to high frequency variations
in the intensity. A quick look at Wikipedia will guide you to a number of other edge detection
techniques that may be more accurate. It is also possible to reduce the amount of high
frequency variation by adding a “blur pass” between the render and edge detection passes.
The “blur pass” will smooth out the high frequency fluctuations and may improve the results
of the edge detection pass.

Optimization techniques
The technique discussed here requires eight texture fetches. Texture accesses can be
somewhat slow, and reducing the number of accesses can result in substantial speed
improvements. Chapter 24 of GPU Gems: Programming Techniques, Tips and Tricks for
Real-Time Graphics, edited by Randima Fernando (Addison-Wesley Professional 2004), has
an excellent discussion of ways to reduce the number of texture fetches in a filter operation
by making use of so-called “helper” textures.

implementing HDR lighting with tone mapping

When rendering for most output devices (monitors or televisions), the device only supports
a typical color precision of 8 bits per color component, or 24 bits per pixel. Therefore, for a
given color component, we’re limited to a range of intensities between 0 and 255. Internally,
OpenGL uses floating-point values for color intensities, providing a wide range of both values
and precision. These are eventually converted to 8 bit values by mapping the floating-point
range [0.0, 1.0] to the range of an unsigned byte [0, 255] before rendering.
Real scenes, however, have a much wider range of luminance. For example, light sources that
are visible in a scene, or direct reflections of them, can be hundreds to thousands of times
brighter than the objects that are illuminated by the source. When we’re working with 8 bits
per channel, or the floating-point range [0.0, -1.0], we can’t represent this range of intensities.
If we decide to use a larger range of floating point values, we can do a better job of internally
representing these intensities, but in the end, we still need to compress down to the 8-bit range.

The process of computing the lighting/shading using a larger dynamic range is often referred
to as High Dynamic Range rendering (HDR rendering). Photographers are very familiar with this concept. When a photographer wants to capture a larger range of intensities than would normally be possible in a single exposure, he/she might take several images with different exposures to capture a wider range of values. This concept, called High Dynamic Range imaging (HDR imaging), is very similar in nature to the concept of HDR rendering.
A post-processing pipeline that includes HDR is now considered a fundamentally essential part of any game engine.
Tone mapping is the process of taking a wide dynamic range of values and compressing them into a smaller range that is appropriate for the output device. In computer graphics, generally, tone mapping is about mapping to the 8-bit range from some arbitrary range of values. The goal is to maintain the dark and light parts of the image so that both are visible, and neither is completely “washed out”.

For example, a scene that includes a bright light source might cause our shading model to produce intensities that are greater than 1.0. If we were to simply send that to the output device, anything greater than 1.0 would be clamped to 255, and would appear white. The
result might be an image that is mostly white, similar to a photograph that is over exposed. Or, if we were to linearly compress the intensities to the [0, 255] range, the darker parts might be too dark, or completely invisible. With tone mapping, we want to maintain the brightness of the light source, and also maintain detail in the darker areas.

如果是线性压缩的话，那么太亮的是白色，太暗的是黑色，而tone mapping能够保留亮的部分，同时黑的部分也保留。

This description just scratches the surface when it comes to tone mapping and HDR rendering/imaging. For more
details, I recommend the book High Dynamic Range Imaging by Reinhard et al.

The mathematical function used to map from one dynamic range to a smaller range is called the Tone Mapping Operator (TMO). These generally come in two “flavors”, local operators and global operators. A local operator determines the new value for a given pixel by using its current value and perhaps the value of some nearby pixels. A global operator needs some information about the entire image, in order to do its work. For example, it might need to have the overall average luminance of all pixels in the image. Other global operators use a histogram of luminance values over the entire image to help fine-tune the mapping. In this recipe, we’ll use a simple global operator that is described in the book Real Time Rendering. This operator uses the log-average luminance of all pixels in the image. The logaverage is determined by taking the logarithm of the luminance and averaging those values,
then converting back, as shown in the following equation:

Lw(x, y) is the luminance of the pixel at (x, y). The 0.0001 term is included in order to avoid
taking the logarithm of zero for black pixels. This log-average is then used as part of the tone
mapping operator shown as follows:.

The a term in this equation is the key. It acts in a similar way to the exposure level in a camera.
The typical values for a range from 0.18 to 0.72. Since this tone mapping operator compresses
the dark and light values a bit too much, we’ll use a modification of the previous equation that
doesn’t compress the dark values as much, and includes a maximum luminance (Lwhite), a
configurable value that helps to reduce some of the extremely bright pixels.

this is the tone mapping operator that we will use in this example.
we will render the scene to a high-resolution buffer, compute the log-average luminance, and then apply the previous tone-mapping operator in a second pass.
However, there’s one more detail that we need to deal with before we can start implementing.
The previous equations all deal with luminance. Starting with an RGB value, we can compute
its luminance, but once we modify the luminance, how do we modify the RGB components to
reflect the new luminance, but without changing the hue (or chromaticity)?

The chromaticity is the perceived color, independent of the
brightness of that color. For example, grey and white are two
brightness levels for the same color.

The solution involves switching color spaces. If we convert the scene to a color space that
separates out the luminance from the chromaticity, then we can change the luminance value
independently. The CIE XYZ color space has just what we need. The CIE XYZ color space was
designed so that the Y component describes the luminance of the color, and the chromaticity
can be determined by two derived parameters (x and y). The derived color space is called the
CIE xyY space, and is exactly what we’re looking for. The Y component contains the luminance
and the x and y components contain the chromaticity. By converting to the CIE xyY space,
we’ve factored out the luminance from the chromaticity allowing us to change the luminance
without affecting the perceived color.
So the process involves converting from RGB to CIE XYZ, then converting to CIE xyY, modifying
the luminance and reversing the process to get back to RGB. To convert from RGB to CIE XYZ
(and vice-versa) can be described as a transformation matrix (refer to the code or the See also
section for the matrix).
The conversion from XYZ to xyY involves the following:

Finally, converting from xyY back to XYZ is done using the following equations:

The following images show an example of the results of this tone mapping operator. The left
image shows the scene rendered without any tone mapping. The shading was deliberately
calculated with a wide dynamic range using three strong light sources. The scene appears
“blown out” because any values that are greater than 1.0 simply get clamped to the maximum
intensity. The image on the right uses the same scene and the same shading, but with the
previous tone mapping operator applied. Note the recovery of the specular highlights from the
“blown-out” areas on the sphere and teapot.

The steps involved are the following:

1. Render the scene to a high-resolution texture.
2. Compute the log-average luminance (on the CPU).
3. Render a screen-filling quad to execute the fragment shader for each screen pixel. In
   the fragment shader, read from the texture created in step 1, apply the tone mapping
   operator, and send the results to the screen.
   To get set up, create a high-res texture (using GL_RGB32F or similar format) attached to a
   framebuffer with a depth attachment. Set up your fragment shader with a subroutine for each
   pass. The vertex shader can simply pass through the position and normal in eye coordinates.

To implement HDR tone mapping, we’ll use the following steps:

1. In the first pass we want to just render the scene to the high-resolution texture. Bind
   to the framebuffer that has the texture attached and render the scene normally.
   Apply whatever shading equation strikes your fancy.
2. Compute the log average luminance of the pixels in the texture. To do so, we’ll pull
   the data from the texture and loop through the pixels on the CPU side. We do this on
   the CPU for simplicity, a GPU implementation, perhaps with a compute shader, would
   be faster.

GLfloat *texData = new GLfloat[width*height*3];
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, hdrTex);
glGetTexImage(GL_TEXTURE_2D, 0, GL_RGB, GL_FLOAT, texData);
float sum = 0.0f;
int size = width*height;
for( int i = 0; i < size; i++ ) {
float lum = computeLuminance(
texData[i*3+0], texData[i*3+1], texData[i*3+2]));
sum += logf( lum + 0.00001f );
}
delete [] texData;
float logAve = expf( sum / size );

3. Set the AveLum uniform variable using logAve. Switch back to the default frame
   buffer, and draw a screen-filling quad. In the fragment shader, apply the tone
   mapping operator to the values from the texture produced in step 1.

// Retrieve high-res color from texture
vec4 color = texture( HdrTex, TexCoord );
// Convert to XYZ
vec3 xyzCol = rgb2xyz * vec3(color);
// Convert to xyY
float xyzSum = xyzCol.x + xyzCol.y + xyzCol.z;
vec3 xyYCol = vec3(0.0);
if( xyzSum > 0.0 ) // Avoid divide by zero
xyYCol = vec3( xyzCol.x / xyzSum,
xyzCol.y / xyzSum, xyzCol.y);
// Apply the tone mapping operation to the luminance
// (xyYCol.z or xyzCol.y)
float L = (Exposure * xyYCol.z) / AveLum;

L = (L * ( 1 + L / (White * White) )) / ( 1 + L );
// Using the new luminance, convert back to XYZ
if( xyYCol.y > 0.0 ) {
xyzCol.x = (L * xyYCol.x) / (xyYCol.y);
xyzCol.y = L;
xyzCol.z = (L * (1 - xyYCol.x - xyYCol.y))/xyYCol.y;
}
// Convert back to RGB and send to output buffer
FragColor = vec4( xyz2rgb * xyzCol, 1.0);

In the first step, we render the scene to an HDR texture. In step 2, we compute the log-average
luminance by retrieving the pixels from the texture and doing the computation on the CPU
(OpenGL side).
In step 3, we render a single screen-filling quad to execute the fragment shader for each
screen pixel. In the fragment shader, we retrieve the HDR value from the texture and apply
the tone-mapping operator. There are two “tunable” variables in this calculation. The variable
Exposure corresponds to the a term in the tone mapping operator, and the variable White
corresponds to Lwhite. For the previous image, we used values of 0.35 and 0.928 respectively.

Tone mapping is not an exact science. Often, it is a process of experimenting with the
parameters until you find something that works well and looks good.
We could improve the efficiency of the previous technique by implementing step 2 on the GPU
using compute shaders (refer to Chapter 10, Using Compute Shaders) or some other clever
technique. For example, we could write the logarithms to a texture, then iteratively downsample
the full frame to a 1 x 1 texture. The final result would be available in that single pixel. However,
with the flexibility of the compute shader, we could optimize this process even more.