Efficient Gradient-domain Compositing
This website documents a project carried out in the course of the seminar "Processing Digital Camera Images" held by the Computer Graphics Group at Technische Universität Berlin in winter term 2011/2012.
Our goal was to implement an efficient gradient-domain compositing algorithm based on quadtrees (Agarwala [AGA07]) and compare it to regular poisson blending as introduced by Pérez et al. [PER03].
We outline the general concept of gradient domain editing, followed by a description of a computationally efficient way to approximate the solution of the poisson equation for image compositing applications. Furthermore we give an overview of our implementation and present our results. Finally a qualitative analysis of the approximation error in our results is given.
2. Gradient-domain Editing
Simple color compositing has the deficiency of discontinuities along transitions between two images (i.e. visible seams as depicted in Figure 2.1).
Gradient domain blending addresses that by forming composites with respect to the spacial gradients of an image rather than its actual color values. Thus, images are edited by altering
their directional changes in color intensity, which are small if neighboring pixels are similar and vice versa.
Figure 2.1: Color compositing (left) vs. gradient-domain compositing (right)
2.1 Integrability Condition
If the gradient of an image I is edited, the result is usually no gradient anymore, but only a general vector field that is not re-integrable. This is because in most cases the changed gradient no longer satisfies the necessary integrability
condition of path independence. A vector field is path independent (or conservative) if it has a vanishing
curl as illustrated in Figure 2.2. In other words the cumulated gradients along any closed loop need to equal zero:
Figure 2.2: If a vector field has a vanishing curl, it is path independent and therefore a gradient field.
2.1 Least-squares Minimization
The idea is to recover an image that is best reflected by this edited gradient. Since the gradient provides us with linear constraints for every pixel in every color channel (see Figure 2.3), we can formulate a (overdetermined) system of
Figure 2.3: The edited gradient provides linear constraints for every pixel in every color channel that a solution needs to satisfy as good as possible.
We set the desired gradient field Ax equal to the constraints defined by this vector field
b, where A is the representation of the gradient as a linear transformation of the original image
x. The result is one large linear system per color channel with one variable per pixel:
As stated before, there may not be an image that satisfies all constraints simultaneously, so we solve the image that best satisfies them in the least-square sense by forming the normal equations:
Once we solve one system of linear equations for every color channel, we obtain
x, which is our desired gradient-domain composite. Unfortunately this problem does not scale, because solving a system of linear equations requires storing the whole vectors
x and b in core memory. This is a limitation, because even for relatively small images the memory requirements become high.
This gradient-domain image editing technique was originally described by Pérez et al. [PER03].
2.1 TalksFor further information please refer to our midterm presentations on "Solving the Poisson equation" and "Applications of Gradient Domain Imaging":
3. Efficient Gradient-domain compositing
The more efficient solver is based on the key in sight that the offset
xδ between the gradient-domain composite x and the color composite
x0 is only complex in areas near seams and very smooth otherwise (see Figure 3.1). Instead of solving for
x, we solve for the residual xδ:
Our desired solution x is split up into x0 + xδ in the system of linear equations that we obtained through our vector field constraints. The system is then rearranged and the normal equations are formed
in order to solve it in the least square sense.
Figure 3.1: Difference map xδ between a gradient-domain composite x and the corresponding color composite x0 and
On the right side of the equation are no longer the desired gradients
b, but the difference between them and the gradients of the color composite (b - Ax0). As depicted in Figure 3.2 they are only going to be non-zero for pixels that are bordering a seam. In all other areas the desired gradient is
created from the same source that the color composite copied its values from and will therefore by zero.
Figure 3.2: For pixels away from seams the gradient within an image region is copied from the same source as the gradient from the color composite. Their difference is therefore only non-zero for pixels bordering a seam.
We take advantage of this characteristic to create a low memory efficient solver according to Agarwala [AGA07]. Instead of using one variable per pixel, fewer
variables are used in smooth areas. The full solution is then interpolated from that.
3.1 Reducing the Problem Size
The choice of variables is determined by quadtrees that are maximally subdivided along the seams. The variables are placed at node corners so that we have fewer of them. To ensure a gradual reduction in resolution from the seams to the interior,
we restrict our quadtree by defining that two neighboring nodes can not defer in size more than a factor of two. Figure 3.3 shows a regular quadtree that only subdivides the space in seam-areas versus the restricted one as implemented by us.
Figure 3.3: A regular quadtree which subdivides the image along seams (left) and a constrained quadtree where neighboring quads can not defer in size more than a factor of two (right).
In order to obtain square nodes, the quadtree is computed on the smallest square with a side length of a power of two that fits our image. Consequently the smallest leaf node is going to be exactly one pixel in size. To be able to crop the
quadtree decomposition back to the original size of the image without cutting any quads, it is also maximally subdivided along the bottom and right side (see Figure 3.4).
Figure 3.4: The quadtree decomposition is computed on the smallest square with a side length of a power of two that fits the composite image.
3.1 Interpolation to Full Space
The quadtree decomposition yields all the pixels that are relevant for a good approximation of our desired solution, so that the rest of the image can be discarded. This reduced space is obviously much smaller than the full image so that
the corresponding system of linear equations can be solved a lot quicker.
Figure 3.5: Bilinear interpolation of the underlying image pixels from the calculated values in the quadtree.
As stated before, we can convert between both spaces using simple bilinear interpolation (encoded in the transition matrix S as depicted in Figure 3.5). The only specialty about this interpolation is that variables that lie along a T-junctions
(where nodes of different sizes abut) are not computed directly but interpolated from the variables that lie along the greater side of the T. This procedure ensures a continuous solution.Note that this interpolation needs to be encoded into the system of linear
equations so that the new matrix AS looses the characteristical structure of a Laplacian matrix.
3.1 Final PresentationFor further information please refer to our final presentation:
We implemented our project using
MATLAB. It features two poisson solvers: one that solves for the full image and one that approximates the result as described above. Also, a graphical user interface is provided that allows the selection of an image patch from a source image and the insertion
into a target image. Because our application focuses on inserting closed image regions of one image into another, we use Dirichlet as opposed to Neumann boundary conditions like proposed by Agarwala [AGA07].
Consequently the target image is constrained to stay unchanged so that the blending process only affects the source image.
Note: Since MATLAB is an interpreted programming language the execution of commands is much slower than in languages that are compiled to machine code. This is why algorithms that are implemented manually have a longer running
time than build-in ones. Therefore the performance gains of our fast poisson solver are distorted in terms of total running time since it contains a lot of custom code. To show how significantly faster the approximated solution can be calculated, we only considered
the solving time and needed number of variables to evaluate our results in the overview below.
4.1 Source Code
In the following some results are presented that we computed using our application. At the end of this section the gradient-domain composites for every result set are compared to their corresponding full space calculations to evaluate the
|Result Set 1 (Balloon): Color composite, full space gradient-domain composite, reduced space gradient-domain composite, quadtree, full space residual, reduced space residual (from left to right, top to bottom)|
|Result Set 2 (Polar Bear): Color composite, full space gradient-domain composite, reduced space gradient-domain composite, quadtree, full space residual, reduced space residual (from left to right, top to bottom)|
|Result Set 3 (Hummingbird): Color composite, full space gradient-domain composite, reduced space gradient-domain composite, quadtree, full space residual, reduced space residual (from left to right, top to bottom)|
|Test Set||Variables (Full)||Variables (Reduced)||Time (Full)||Time (Reduced)||Error* (Ø**/Max)|
|Polar Bear||506700||9186||6.75s||0.06s||1.2941e-06/ 5|
* Measured in gray level values in the range [0, 255]
** Mean of absolute per-pixel error in source image patch
Our implementation of the efficient gradient-domain compositing algorithm produces approximations that are visually indistinguishable from the full space compositions. Because the problem size is reduced by more than 98 percent in our example
images, the efficient solver needs a lot less memory and is faster than the regular one.
As opposed to other fast Poisson solvers, this approximation is the only solution that actually reduces the scale of the problem, instead of optimizing the solving process itself.
- [AGA07] Agarwala, A. (2007). Efficient gradient-domain compositing using quadtrees. ACM Transactions on Graphics- Proceedings of SIGGRAPH 2007. New York, New York, USA. [
- [FAR09] Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., Lischinski, D. (2009). Coordinates for instant image cloning. ACM SIGGRAPH 2009 Papers- SIGGRAPH ’09. New York, New York, USA. [
- [PER03] Pérez, P., Gangnet, M. & Blake, A. (2003). Poisson image editing. ACM SIGGRAPH 2003 Papers- SIGGRAPH ’03, 313. New York, New York, USA. [ pdf ]
Stefan Chmiela - stefan (at) chmiela.com
Andreas Langenhagen - andreas.langenhagen (at) dai-labor.de