Efficient Gradient-domain Compositing

Efficient Gradient-domain Compositing

from: http://cybertron.cg.tu-berlin.de/pdci11ws/gdi/

1. Introduction

This website documents a project carried out in the course of the seminar "Processing Digital Camera Images" held by the Computer Graphics Group at Technische Universität Berlin in winter term 2011/2012.
Our goal was to implement an efficient gradient-domain compositing algorithm based on quadtrees (Agarwala [AGA07]) and compare it to regular poisson blending as introduced by Pérez et al. [PER03].
We outline the general concept of gradient domain editing, followed by a description of a computationally efficient way to approximate the solution of the poisson equation for image compositing applications. Furthermore we give an overview of our implementation and present our results. Finally a qualitative analysis of the approximation error in our results is given.

2. Gradient-domain Editing

Simple color compositing has the deficiency of discontinuities along transitions between two images (i.e. visible seams as depicted in Figure 2.1). Gradient domain blending addresses that by forming composites with respect to the spacial gradients of an image rather than its actual color values. Thus, images are edited by altering their directional changes in color intensity, which are small if neighboring pixels are similar and vice versa.

Figure 2.1: Color compositing (left) vs. gradient-domain compositing (right)

2.1 Integrability Condition

If the gradient of an image I is edited, the result is usually no gradient anymore, but only a general vector field that is not re-integrable. This is because in most cases the changed gradient no longer satisfies the necessary integrability condition of path independence. A vector field is path independent (or conservative) if it has a vanishing curl as illustrated in Figure 2.2. In other words the cumulated gradients along any closed loop need to equal zero:

Figure 2.2: If a vector field has a vanishing curl, it is path independent and therefore a gradient field.

2.1 Least-squares Minimization

The idea is to recover an image that is best reflected by this edited gradient. Since the gradient provides us with linear constraints for every pixel in every color channel (see Figure 2.3), we can formulate a (overdetermined) system of linear equations.

Figure 2.3: The edited gradient provides linear constraints for every pixel in every color channel that a solution needs to satisfy as good as possible.

We set the desired gradient field Ax equal to the constraints defined by this vector field b, where A is the representation of the gradient as a linear transformation of the original image x. The result is one large linear system per color channel with one variable per pixel:

As stated before, there may not be an image that satisfies all constraints simultaneously, so we solve the image that best satisfies them in the least-square sense by forming the normal equations:

Once we solve one system of linear equations for every color channel, we obtain x, which is our desired gradient-domain composite. Unfortunately this problem does not scale, because solving a system of linear equations requires storing the whole vectors x and b in core memory. This is a limitation, because even for relatively small images the memory requirements become high.

This gradient-domain image editing technique was originally described by Pérez et al. [PER03].

2.1 Talks

For further information please refer to our midterm presentations on "Solving the Poisson equation" and "Applications of Gradient Domain Imaging":

• Talk - The Poisson Equation [ pdf ]
  
  
• Talk - Applications of Gradient Domain Imaging [ pdf ]
  
  

3. Efficient Gradient-domain compositing

The more efficient solver is based on the key in sight that the offset x_δ between the gradient-domain composite x and the color composite x₀ is only complex in areas near seams and very smooth otherwise (see Figure 3.1). Instead of solving for x, we solve for the residual x_δ:

Our desired solution x is split up into x₀ + x_δ in the system of linear equations that we obtained through our vector field constraints. The system is then rearranged and the normal equations are formed in order to solve it in the least square sense.

Figure 3.1: Difference map xδ between a gradient-domain composite x and the corresponding color composite x0 and

On the right side of the equation are no longer the desired gradients b, but the difference between them and the gradients of the color composite (b - Ax₀). As depicted in Figure 3.2 they are only going to be non-zero for pixels that are bordering a seam. In all other areas the desired gradient is created from the same source that the color composite copied its values from and will therefore by zero.

Figure 3.2: For pixels away from seams the gradient within an image region is copied from the same source as the gradient from the color composite. Their difference is therefore only non-zero for pixels bordering a seam.

We take advantage of this characteristic to create a low memory efficient solver according to Agarwala [AGA07]. Instead of using one variable per pixel, fewer variables are used in smooth areas. The full solution is then interpolated from that.

3.1 Reducing the Problem Size

The choice of variables is determined by quadtrees that are maximally subdivided along the seams. The variables are placed at node corners so that we have fewer of them. To ensure a gradual reduction in resolution from the seams to the interior, we restrict our quadtree by defining that two neighboring nodes can not defer in size more than a factor of two. Figure 3.3 shows a regular quadtree that only subdivides the space in seam-areas versus the restricted one as implemented by us.

Figure 3.3: A regular quadtree which subdivides the image along seams (left) and a constrained quadtree where neighboring quads can not defer in size more than a factor of two (right).

In order to obtain square nodes, the quadtree is computed on the smallest square with a side length of a power of two that fits our image. Consequently the smallest leaf node is going to be exactly one pixel in size. To be able to crop the quadtree decomposition back to the original size of the image without cutting any quads, it is also maximally subdivided along the bottom and right side (see Figure 3.4).

Figure 3.4: The quadtree decomposition is computed on the smallest square with a side length of a power of two that fits the composite image.

3.1 Interpolation to Full Space

The quadtree decomposition yields all the pixels that are relevant for a good approximation of our desired solution, so that the rest of the image can be discarded. This reduced space is obviously much smaller than the full image so that the corresponding system of linear equations can be solved a lot quicker.

Figure 3.5: Bilinear interpolation of the underlying image pixels from the calculated values in the quadtree.

As stated before, we can convert between both spaces using simple bilinear interpolation (encoded in the transition matrix S as depicted in Figure 3.5). The only specialty about this interpolation is that variables that lie along a T-junctions (where nodes of different sizes abut) are not computed directly but interpolated from the variables that lie along the greater side of the T. This procedure ensures a continuous solution.Note that this interpolation needs to be encoded into the system of linear equations so that the new matrix AS looses the characteristical structure of a Laplacian matrix.

For the sake of completeness it should be noted that a similar approach using mean-value coordinates instead of quadtrees was developed by [FAR09].

3.1 Final Presentation

For further information please refer to our final presentation:

• Talk - Gradient-domain compositing [ pdf ]
  
  

4. Implementation

We implemented our project using MATLAB. It features two poisson solvers: one that solves for the full image and one that approximates the result as described above. Also, a graphical user interface is provided that allows the selection of an image patch from a source image and the insertion into a target image. Because our application focuses on inserting closed image regions of one image into another, we use Dirichlet as opposed to Neumann boundary conditions like proposed by Agarwala [AGA07]. Consequently the target image is constrained to stay unchanged so that the blending process only affects the source image.

Note: Since MATLAB is an interpreted programming language the execution of commands is much slower than in languages that are compiled to machine code. This is why algorithms that are implemented manually have a longer running time than build-in ones. Therefore the performance gains of our fast poisson solver are distorted in terms of total running time since it contains a lot of custom code. To show how significantly faster the approximated solution can be calculated, we only considered the solving time and needed number of variables to evaluate our results in the overview below.

4.1 Source Code

• Matlab implementation [ zip ]
  
  

5. Results

In the following some results are presented that we computed using our application. At the end of this section the gradient-domain composites for every result set are compared to their corresponding full space calculations to evaluate the approximation quality.

Result Set 1 (Balloon): Color composite, full space gradient-domain composite, reduced space gradient-domain composite, quadtree, full space residual, reduced space residual (from left to right, top to bottom)

Result Set 2 (Polar Bear): Color composite, full space gradient-domain composite, reduced space gradient-domain composite, quadtree, full space residual, reduced space residual (from left to right, top to bottom)

Result Set 3 (Hummingbird): Color composite, full space gradient-domain composite, reduced space gradient-domain composite, quadtree, full space residual, reduced space residual (from left to right, top to bottom)

Test Set	Variables (Full)	Variables (Reduced)	Time (Full)	Time (Reduced)	Error* (Ø**/Max)
Balloon	506700	8175	4.04s	0.04s	1.8937e-06/ 3
Polar Bear	506700	9186	6.75s	0.06s	1.2941e-06/ 5
Hummingbird	520200	10029	4.32s	0.06s	8.6911e-07/ 4

* Measured in gray level values in the range [0, 255]
** Mean of absolute per-pixel error in source image patch

6. Conclusion

Our implementation of the efficient gradient-domain compositing algorithm produces approximations that are visually indistinguishable from the full space compositions. Because the problem size is reduced by more than 98 percent in our example images, the efficient solver needs a lot less memory and is faster than the regular one.

As opposed to other fast Poisson solvers, this approximation is the only solution that actually reduces the scale of the problem, instead of optimizing the solving process itself.

References

• [AGA07] Agarwala, A. (2007). Efficient gradient-domain compositing using quadtrees. ACM Transactions on Graphics- Proceedings of SIGGRAPH 2007. New York, New York, USA. [ web ]
  
  
• [FAR09] Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., Lischinski, D. (2009). Coordinates for instant image cloning. ACM SIGGRAPH 2009 Papers- SIGGRAPH ’09. New York, New York, USA. [ pdf ]
  
  
• [PER03] Pérez, P., Gangnet, M. & Blake, A. (2003). Poisson image editing. ACM SIGGRAPH 2003 Papers- SIGGRAPH ’03, 313. New York, New York, USA. [ pdf ]

Contact

Stefan Chmiela - stefan (at) chmiela.com

Andreas Langenhagen - andreas.langenhagen (at) dai-labor.de