Implement a BlockingQueue using conditon variable

One problem that comes up time and again with multi-threaded code is how to transfer data from one thread to another. Forexample, one common way to parallelize a serial algorithm is to split it into independent chunks and make a pipeline — eachstage in the pipeline can be run on a separate thread, and each stage adds the data to the input queue for the next stage when it'sdone. For this to work properly, the input queue needs to be written so that data can safely be added by one thread and removed byanother thread without corrupting the data structure.

Basic Thread Safety with a Mutex

The simplest way of doing this is just to put wrap a non-thread-safe queue, and protect it with a mutex (the examples use thetypes and functions from the upcoming 1.35 release of Boost):

template<typename Data>
class concurrent_queue
{
private:
    std::queue<Data> the_queue;
    mutable boost::mutex the_mutex;
public:
    void push(const Data& data)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        the_queue.push(data);
    }

    bool empty() const
    {
        boost::mutex::scoped_lock lock(the_mutex);
        return the_queue.empty();
    }

    Data& front()
    {
        boost::mutex::scoped_lock lock(the_mutex);
        return the_queue.front();
    }
    
    Data const& front() const
    {
        boost::mutex::scoped_lock lock(the_mutex);
        return the_queue.front();
    }

    void pop()
    {
        boost::mutex::scoped_lock lock(the_mutex);
        the_queue.pop();
    }
};

This design is subject to race conditions between calls to empty, front and pop if thereis more than one thread removing items from the queue, but in a single-consumer system (as being discussed here), this is not aproblem. There is, however, a downside to such a simple implementation: if your pipeline stages are running on separate threads,they likely have nothing to do if the queue is empty, so they end up with a wait loop:

    while(some_queue.empty())
    {
        boost::this_thread::sleep(boost::posix_time::milliseconds(50));
    }

Though the sleep avoids the high CPU consumption of a direct busy wait, there are still some obvious downsides tothis formulation. Firstly, the thread has to wake every 50ms or so (or whatever the sleep period is) in order to lock the mutex,check the queue, and unlock the mutex, forcing a context switch. Secondly, the sleep period imposes a limit on how fast the threadcan respond to data being added to the queue — if the data is added just before the call to sleep, the threadwill wait at least 50ms before checking for data. On average, the thread will only respond to data after about half the sleep time(25ms here).

Waiting with a Condition Variable

As an alternative to continuously polling the state of the queue, the sleep in the wait loop can be replaced with a conditionvariable wait. If the condition variable is notified in push when data is added to an empty queue, then the waitingthread will wake. This requires access to the mutex used to protect the queue, so needs to be implemented as a member function ofconcurrent_queue:

template<typename Data>
class concurrent_queue
{
private:
    boost::condition_variable the_condition_variable;
public:
    void wait_for_data()
    {
        boost::mutex::scoped_lock lock(the_mutex);
        while(the_queue.empty())
        {
            the_condition_variable.wait(lock);
        }
    }
    void push(Data const& data)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        bool const was_empty=the_queue.empty();
        the_queue.push(data);
        if(was_empty)
        {
            the_condition_variable.notify_one();
        }
    }
    // rest as before
};

There are three important things to note here. Firstly, the lock variable is passed as a parameter towait — this allows the condition variable implementation to atomically unlock the mutex and add the thread to thewait queue, so that another thread can update the protected data whilst the first thread waits.

Secondly, the condition variable wait is still inside a while loop — condition variables can be subject tospurious wake-ups, so it is important to check the actual condition being waited for when the call to waitreturns.

Be careful when you notify

Thirdly, the call to notify_one comes after the data is pushed on the internal queue. This avoids thewaiting thread being notified if the call to the_queue.push throws an exception. As written, the call tonotify_one is still within the protected region, which is potentially sub-optimal: the waiting thread might wake upimmediately it is notified, and before the mutex is unlocked, in which case it will have to block when the mutex is reacquired onthe exit from wait. By rewriting the function so that the notification comes after the mutex is unlocked, thewaiting thread will be able to acquire the mutex without blocking:

template<typename Data>
class concurrent_queue
{
public:
    void push(Data const& data)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        bool const was_empty=the_queue.empty();
        the_queue.push(data);

        lock.unlock(); // unlock the mutex

        if(was_empty)
        {
            the_condition_variable.notify_one();
        }
    }
    // rest as before
};

Reducing the locking overhead

Though the use of a condition variable has improved the pushing and waiting side of the interface, the interface for the consumerthread still has to perform excessive locking: wait_for_data, front and pop all lock themutex, yet they will be called in quick succession by the consumer thread.

By changing the consumer interface to a single wait_and_pop function, the extra lock/unlock calls can be avoided:

template<typename Data>
class concurrent_queue
{
public:
    void wait_and_pop(Data& popped_value)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        while(the_queue.empty())
        {
            the_condition_variable.wait(lock);
        }
        
        popped_value=the_queue.front();
        the_queue.pop();
    }

    // rest as before
};

Using a reference parameter to receive the result is used to transfer ownership out of the queue in order to avoid the exceptionsafety issues of returning data by-value: if the copy constructor of a by-value return throws, then the data has been removed fromthe queue, but is lost, whereas with this approach, the potentially problematic copy is performed prior to modifying the queue (seeHerb Sutter's Guru Of The Week #8 for a discussion of the issues). This does, ofcourse, require that an instance Data can be created by the calling code in order to receive the result, which is notalways the case. In those cases, it might be worth using something like boost::optional to avoid this requirement.

Handling multiple consumers

As well as removing the locking overhead, the combined wait_and_pop function has another benefit — itautomatically allows for multiple consumers. Whereas the fine-grained nature of the separate functions makes them subject to raceconditions without external locking (one reason why the authors of the SGISTL advocate against making things like std::vector thread-safe — you need external locking to do many commonoperations, which makes the internal locking just a waste of resources), the combined function safely handles concurrent calls.

If multiple threads are popping entries from a full queue, then they just get serialized inside wait_and_pop, andeverything works fine. If the queue is empty, then each thread in turn will block waiting on the condition variable. When a newentry is added to the queue, one of the threads will wake and take the value, whilst the others keep blocking. If more than onethread wakes (e.g. with a spurious wake-up), or a new thread calls wait_and_pop concurrently, the whileloop ensures that only one thread will do the pop, andthe others will wait.

Update: As commenter David notes below, using multiple consumers does have one problem: if there are severalthreads waiting when data is added, only one is woken. Though this is exactly what you want if only one item is pushed onto thequeue, if multiple items are pushed then it would be desirable if more than one thread could wake. There are two solutions to this:use notify_all() instead of notify_one() when waking threads, or to call notify_one()whenever any data is added to the queue, even if the queue is not currently empty. If all threads are notified then the extrathreads will see it as a spurious wake and resume waiting if there isn't enough data for them. If we notify with everypush() then only the right number of threads are woken. This is my preferred option: condition variable notify callsare pretty light-weight when there are no threads waiting. The revised code looks like this:

template<typename Data>
class concurrent_queue
{
public:
    void push(Data const& data)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        the_queue.push(data);
        lock.unlock();
        the_condition_variable.notify_one();
    }
    // rest as before
};

There is one benefit that the separate functions give over the combined one — the ability to check for an empty queue, anddo something else if the queue is empty. empty itself still works in the presence of multiple consumers, but the valuethat it returns is transitory — there is no guarantee that it will still apply by the time a thread callswait_and_pop, whether it was true or false. For this reason it is worth adding an additionalfunction: try_pop, which returns true if there was a value to retrieve (in which case it retrieves it), orfalse to indicate that the queue was empty.

template<typename Data>
class concurrent_queue
{
public:
    bool try_pop(Data& popped_value)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        if(the_queue.empty())
        {
            return false;
        }
        
        popped_value=the_queue.front();
        the_queue.pop();
        return true;
    }

    // rest as before
};

By removing the separate front and pop functions, our simple naive implementation has now become ausable multiple producer, multiple consumer concurrent queue.

The Final Code

Here is the final code for a simple thread-safe multiple producer, multiple consumer queue:

template<typename Data>
class concurrent_queue
{
private:
    std::queue<Data> the_queue;
    mutable boost::mutex the_mutex;
    boost::condition_variable the_condition_variable;
public:
    void push(Data const& data)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        the_queue.push(data);
        lock.unlock();
        the_condition_variable.notify_one();
    }

    bool empty() const
    {
        boost::mutex::scoped_lock lock(the_mutex);
        return the_queue.empty();
    }

    bool try_pop(Data& popped_value)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        if(the_queue.empty())
        {
            return false;
        }
        
        popped_value=the_queue.front();
        the_queue.pop();
        return true;
    }

    void wait_and_pop(Data& popped_value)
    {
        boost::mutex::scoped_lock lock(the_mutex);
        while(the_queue.empty())
        {
            the_condition_variable.wait(lock);
        }
        
        popped_value=the_queue.front();
        the_queue.pop();
    }

};