[Cocos3.0 Tutorial] RenderTexture + Blur

This tutorial will help you to understand how you can use shaders and render textures in cocos2d-x 3.0 and will give you some insights in the underlying math. In one of my projects I faced a necessity to make a blurred version of some background landscape, which composition depends on the screen size, that is clouds are sticked to the top of the screen and ground is sticked to the bottom, so it's not an option to just put a blurred image in the resource folder. The landscape must be composed, drawn in a RenderTexture, then drawn again with blurring shader, saved to disk and used in consequent launches. The image can be pretty big and the blur radius can be pretty big as well, so we need something fast.

This tutorial can be divided into following steps:
1. Diving into math and calculating gaussian weights
2. Creating blur shader
3. Rendering texture and saving it to file
4. Using TextureBlur in a sample program

Let's start. To blur a pixel we just need to calculate a weighted sum of the surrounding pixels, where weights themselves sum up to 1. Another property that would be nice to have is that central weights must be heavier than the side ones. Why Gaussian function is usually picked for making blur effect? Because it has three nice features:
1. It's integral equals 1
2. It's maximum value is in the symmetry point
3. It has the following feature:

Tow-dimensional Gaussian function can be split into a product of two one-dimensional functions. What it will give us? To calculate the color of the blurred pixel with coordinate (i, j) in a straightforward manner we need to sum up all the pixels in range (i-R, i+R)x(j-R, j+R), where R is a blur radius. This results in a nested loop and nested loop means O(n*n) complexity and we really do not want such a thing in a shader. Keeping in mind the 3rd feature of the Gaussian distribution we can split the nested loop in two consequent loops. At first we will do a horizontal blur and then - vertical, thus having O(n) complexity. The magic is that the final result of such simplification won't differ from the one obtained with slow nested loop.

Let's calculate an integral of Gaussian function with sigma = 1/3 and mu = 0 from x = -1 to 1. That will give us 0.9973, almost 1. Let's now use a common technique for numerical integration: we are going to draw 2*R-1 points from -1 to 1, calculate Gaussian function in them, multiple obtained values by 1/(R-1) and sum everything up.

The nice fact is that that sum will equal to something near 1 and the precision will grow together with R. The only problem left to solve is that we want coefficients to sum up exactly to 1, otherwise the blurred image will have some problems with opacity (totally opaque images will become a little bit transparent). This can be guaranteed by calculating the central coefficient as 1 - sum of all the others.

So, to the codes. The idea is to pre-calculcate gaussian coefficients on start, pass them to the shader and do no math other than multiplication inside.

void TextureBlur::calculateGaussianWeights(const int points, float* weights)
{
    float dx = 1.0f/float(points-1);
    float sigma = 1.0f/3.0f;
    float norm = 1.0f/(sqrtf(2.0f*M_PI)*sigma*points);
    float divsigma2 = 0.5f/(sigma*sigma);
    weights[0] = 1.0f;
    for (int i = 1; i < points; i++)
    {
        float x = float(i)*dx;
        weights[i] = norm*expf(-x*x*divsigma2);
        weights[0] -= 2.0f*weights[i];
    }
}

Ok, our next step is to create a blur shader.

#ifdef GL_ES                                                                      
precision mediump float;
#endif                                                                            

varying vec4 v_fragmentColor;                                                     
varying vec2 v_texCoord;                                                          
uniform sampler2D CC_Texture0;

uniform vec2 pixelSize;
uniform int radius;
uniform float weights[64];
uniform vec2 direction;

void main() {
    gl_FragColor = texture2D(CC_Texture0, v_texCoord)*weights[0];
    for (int i = 1; i < radius; i++) {
        vec2 offset = vec2(float(i)*pixelSize.x*direction.x, float(i)*pixelSize.y*direction.y);
        gl_FragColor += texture2D(CC_Texture0, v_texCoord + offset)*weights[i];
        gl_FragColor += texture2D(CC_Texture0, v_texCoord - offset)*weights[i];
    }
}

Besides the standard v_fragmentColor, v_texCoord and CC_Texture0 we have four additional parameters:
1. pixelSize - in GLSL there are no pixels, only decimals, for instance 0 denotes the left or the lower border of the texture and 1 - the right or the upper border. This means we have to know the decimal step to make to the next pixel.
2. radius - our blur radius
3. weights - array of precalculated gaussian weights
4. direction - 2d vector denoting horizontal or vertical blur
The word "uniform" in front of these values mean that they are not going to change during the run. The rest of the shader body is pretty straightforward: take a pixel, accumulate surrounding pixels with some coefficients and voila. I had to hardcode maximum array size to be 64 as I found no way to use a dynamic array as a uniform in shader.

Our next step is to pass desired parameters to the shader, but before writing down a blur GLProgram factory we have to patch GLProgram class a little bit. The problem is with passing a one dimensional array to the shader as a uniform. While there exist corresponding wrappers in Cocos2d for two, three and four arrays, there's no function for a single one. So here we go:

void GLProgram::setUniformLocationWith1fv(GLint location, const GLfloat* floats, unsigned int numberOfArrays)
{
    bool updated =  updateUniformLocation(location, floats, sizeof(float)*numberOfArrays);

    if( updated )
    {
        glUniform1fv( (GLint)location, (GLsizei)numberOfArrays, floats );
    }
}

Now, when everything is on order, we can continue with blur shader factory:

GLProgram* TextureBlur::getBlurShader(Size pixelSize, Point direction, const int radius, float* weights)
{
    GLProgram* blur = new GLProgram();
    std::string blurShaderPath = FileUtils::getInstance()->fullPathForFilename("Shaders/Blur.fsh");
    const GLchar* blur_frag = String::createWithContentsOfFile(blurShaderPath.c_str())->getCString();
    blur->initWithByteArrays(ccPositionTextureColor_vert, blur_frag);
    blur->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_POSITION, GLProgram::VERTEX_ATTRIB_POSITION);
    blur->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_COLOR, GLProgram::VERTEX_ATTRIB_COLOR);
    blur->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_TEX_COORD, GLProgram::VERTEX_ATTRIB_TEX_COORDS);

    blur->link();
    blur->updateUniforms();

    GLuint pixelSizeLoc = glGetUniformLocation(blur->getProgram(), "pixelSize");
    blur->setUniformLocationWith2f(pixelSizeLoc, pixelSize.width, pixelSize.height);
    GLuint directionLoc = glGetUniformLocation(blur->getProgram(), "direction");
    blur->setUniformLocationWith2f(directionLoc, direction.x, direction.y);
    GLuint radiusLoc = glGetUniformLocation(blur->getProgram(), "radius");
    blur->setUniformLocationWith1i(radiusLoc, radius);
    GLuint weightsLoc = glGetUniformLocation(blur->getProgram(), "weights");
    blur->setUniformLocationWith1fv(weightsLoc, weights, radius);

    return blur;
}

Everything is pretty much self-explanatory here. After binding standard attributes, we pass additional our parameters and the blur shader is ready to rock. One may encounter another way of assigning the shader body to GLchar* variable, something like this:

const GLchar* shader = 
#include "Shader.fsh"

And the Shader.fsh looks like

"\n\
#ifdef GL_ES \n\
precision lowp float; \n\
#endif \n\
...

I prefer using String::createWithContentsOfFile because it frees you from necessity to write \n\ at the end of every line which is quite annoying.

The only thing left to do is actually blurring a texture. Our strategy here will be as follows:
1. Create a sprite from the texture passed as a parameter
2. Draw it to a RenderTexture with horizontal blur shader
3. Create a sprite from resulting texture
4. Draw this sprite to a Render texture with vertical shader
5. Save image to file
6. Wait until saving is done, clean up and notify the caller

Everything is fine so far, except for point 6. You surely have already read these charming words: "In Cocos2d-x v3 the renderer is decoupled from scene graph", but what they actually mean? (If you have not, here's a link http://www.cocos2d-x.org/docs/manual/framework/native/renderer/en26). Actually they mean that in our case we cannot do the following:

saveToFile(fileName);
useTheFile(fileName);

If you dive into saveToFile method of RenderTexture, you will notice, that no saving is happening there. Instead a command is created and added to renderer task queue. When it will be executed? in this thread or the other? To make our blurring tool stable and usable we must guarantee that all objects involved in rendering will be alive when the saveToFile command is executed. And when it's done, we want to get a callback in the current thread. Unfortunately there's no callback functionality in RenderTexture, but that's an open source community here, right? Let's patch another class a little bit by modifying one of the existing methods and adding a new one:

bool RenderTexture::saveToFile(const std::string& fileName, Image::Format format, std::function<void()> callback)
{
    saveToFile(fileName, format);
    onSaveToFileCallback = callback;
    return true;
}

void RenderTexture::onSaveToFile(const std::string& filename)
{
    Image *image = newImage(true);
    if (image)
    {
        image->saveToFile(filename.c_str(), true);
    }

    CC_SAFE_DELETE(image);

    if (onSaveToFileCallback) {
        Director::getInstance()->getScheduler()->performFunctionInCocosThread(onSaveToFileCallback);
        onSaveToFileCallback = nullptr;
    }
}

onSaveToFileCallback is std::function added to RenderTexture class declaration and initialized as nullptr in constructor. Ok, that's better. Now, to the main method. We need the following things to be passed as arguments: texture to blur, blur radius, resulting picture file name, a callback to invoke when everything is done and step as an optional parameter, we'll get to it in a matter of seconds.

void TextureBlur::create(Texture2D* target, const int radius, const std::string& fileName, std::function<void()> callback, const int step)
{
    CCASSERT(target != nullptr, "Null pointer passed as a texture to blur");
    CCASSERT(radius <= maxRadius, "Blur radius is too big");
    CCASSERT(radius > 0, "Blur radius is too small");
    CCASSERT(!fileName.empty(), "File name can not be empty");
    CCASSERT(step <= radius/2 + 1 , "Step is too big");
    CCASSERT(step > 0 , "Step is too small");

Assertions in public interfaces are good. No assertions in public interfaces is bad. Assertions ensure that our program won't crash who knows where, but gently drop, providing a hint about what was wrong.

    Size textureSize = target->getContentSizeInPixels();
    Size pixelSize = Size(float(step)/textureSize.width, float(step)/textureSize.height);
    int radiusWithStep = radius/step;

To speed up thing a little bit we can skip some pixels while calculating blur. If the colors in your texture do not change rapidly from pixel to pixel it's quite alright to use every second one, thus reducing the number of calculations twice. Or thrice. But huge steps should be avoided as they will reduce the quality of the final picture straight off.

    float* weights = new float[maxRadius];
    calculateGaussianWeights(radiusWithStep, weights);

    Sprite* stepX = CCSprite::createWithTexture(target);
    stepX->retain();
    stepX->setPosition(Point(0.5f*textureSize.width, 0.5f*textureSize.height));
    stepX->setFlippedY(true);

    GLProgram* blurX = getBlurShader(pixelSize, Point(1.0f, 0.0f), radiusWithStep, weights);
    stepX->setShaderProgram(blurX);

    RenderTexture* rtX = RenderTexture::create(textureSize.width, textureSize.height);
    rtX->retain();
    rtX->begin();
    stepX->visit();
    rtX->end();

    Sprite* stepY = CCSprite::createWithTexture(rtX->getSprite()->getTexture());
    stepY->retain();
    stepY->setPosition(Point(0.5f*textureSize.width, 0.5f*textureSize.height));
    stepY->setFlippedY(true);

    GLProgram* blurY = getBlurShader(pixelSize, Point(0.0f, 1.0f), radiusWithStep, weights);
    stepY->setShaderProgram(blurY);

    RenderTexture* rtY = RenderTexture::create(textureSize.width, textureSize.height);
    rtY->retain();
    rtY->begin();
    stepY->visit();
    rtY->end();

At this point rtY contains a blurred texture that should be saved to file, let's do it:

    auto completionCallback = [rtX, rtY, stepX, stepY, blurX, blurY, callback]()
    {
        stepX->release();
        stepY->release();
        rtX->release();
        rtY->release();
        delete blurX;
        delete blurY;
        callback();
    };

    rtY->saveToFile(fileName, Image::Format::PNG, completionCallback);
}

Here I used a lambda variable - one of the coolest features of C++11. Recitation of the stepX, stepY etc. means that we want these variables to be captured by their value. That means that when we use these variables in lambda, we actually use their local copies. Another option is to capture variables be reference, but in our case these variables will be destroyed at the moment when the callback will be executed, thus causing undefined behavior. In Cocos2d-x sources you can find [&] or [=] designations. They mean, correspondingly, that all variables should be captured by reference or by value. Some of the C++ safety standards recommend to be as explicit as possible when declaring lambda capturing method, which may be different for every variable.

Finally, lets get TextureBlur to work! Some sample drawing in paint, a little bit of additional code to HelloWorld scene and here we go. That's how initial paysage looks like:

Screen Shot 2014-06-12 at 8.40.08.png 962x642 63 KB

And here is it's blurred version:

Screen Shot 2014-06-12 at 8.40.50.png 960x641 141 KB

You can download sources using the link below:
https://cloud.mail.ru/public/3a0279fe2195/TextureBlur.zip113

I've tested this program on Mac, iOS and Android. Feel free to ask questions, point to bugs, etc.

Thanks