Orocos DataPort 解析: orocos lock free data object

关于 lock-free 数据结构，见这个链接：https://blog.csdn.net/gw569453350game/article/details/51579597

Orocos有两种 dataPort (Input, Output)，在 conncet(Output, Input) 之后，往 Output 中写的数据能够传递到 Input 中并读取, 一个OutputPort可以连接多个InputPort(类似于ROS中的Pub&Sub)，这极大地提高了模块化设计的数据交换。

其中能够实现数据交换的关键是 ChannelDataElement 数据结构，在 connect 两个端口的同时，会 new 一个 ChannelDataElement 数据结构，然后在两个DataPort中分别保存其指针，这个 ChannelDataElement 承担着数据中转站的角色，有 write 和 read 方法分别供 OutputPort 和 InputPort 使用。而在其中起到关键作用的成员变量是 DataObject 对象，它可以是 DATA，BUFFER，CIRCULAR_BUFFER 三种类型之一，提供暂存数据的作用。

ChannelDataElement 数据结构的代码如下：

template<typename T>
class ChannelDataElement : public base::ChannelElement<T>
{
    bool written, mread;
    typename base::DataObjectInterface<T>::shared_ptr data;

public:
    typedef typename base::ChannelElement<T>::param_t param_t;
    typedef typename base::ChannelElement<T>::reference_t reference_t;

    ChannelDataElement(typename base::DataObjectInterface<T>::shared_ptr sample)
        : written(false), mread(false), data(sample) {}

    /** Update the data sample stored in this element.
     * It always returns true. */
    virtual bool write(param_t sample)
    {
        data->Set(sample);
        written = true;
        mread = false;
        return this->signal();
    }

    /** Reads the last sample given to write()
     *
     * @return false if no sample has ever been written, true otherwise
     */
    virtual FlowStatus read(reference_t sample, bool copy_old_data)
    {
        if (written)
        {
            if ( !mread ) {
  data->Get(sample);
                mread = true;
                return NewData;  // 枚举变量
            }

if(copy_old_data)
  data->Get(sample);

            return OldData;  // 枚举变量
        }
        return NoData;  // 枚举变量
    }

    /** Resets the stored sample. After clear() has been called, read()
     * returns false
     */
    virtual void clear()
    {
        written = false;
        mread = false;
        base::ChannelElement<T>::clear();
    }

    virtual bool data_sample(param_t sample)
    {
        data->data_sample(sample);
        return base::ChannelElement<T>::data_sample(sample);
    }

    virtual T data_sample()
    {
        return data->Get();
    }

};

其中数据结构 DataObjectInterface<T> 是结构体 DataObjectLockFree 的基类， 其实现是一个 lock free 的 circular bufer，可以实现 一个 producer + 多个 consumer 的操作（即多个线程同时读，一个线程同时写）
这里还有一个Lock free 的 Mulity Writer Single Reader 数据结构实现（多个线程同时写，一个线程同时读） see link

相比于使用mutex, lock free 的好处就是，当获得这个共享资源的线程被挂起的时候，其他线程照样能够正常读写该资源，可以说是 real-time 的读写。

关于多线程 lock_free 的设计： In general, any time you have a small amount of data protected by a mutex, and you can pack that data entirely into a 32- or 64-bit integer type, you can always convert your mutex-based operations into lock-free RMW operations, no matter what those operations actually do! 但是很多情况用户自定义数据结构(类)有很多数据成员，比64-bit要大得多，而且不一定是 natively aligned，所以读写操作不可能是 atomic 的，存在读写自定义数据的过程中被其他线程中断(如再次进行写操作)的情况。不过，我们可以设计 std::atomic<MyClass*> read_ptr 和 std::atomic<MyClass*> write_ptr，通过 compare_exchange_weak 局部变量来实现安全的指针交换。

Orocos DataObjectLockFree:

namespace RTT
{ namespace base {

    /**
     * @brief This DataObject is a Lock-Free implementation,
     * such that reads and writes can happen concurrently without priority
     * inversions.
     *
     * When there are more writes than reads, the last write will
     * be returned. The internal buffer can get full if too many
     * concurrent reads are taking to long. In that case, each new
     * read will read the element the previous read returned.
     *
     * @verbatim
     * The following Truth table applies when a Low Priority thread is
     * preempted by a High Priority thread :
     *
     *   L\H | Set | Get |
     *   Set | Ok  | Ok  |
     *   Get | Ok  | Ok  |
     *
     * legend : L : Low Priority thread
     *          H : High Priority thread
     *          Blk: Blocks High Priority thread (bad!)
     *          internal::NA : Not allowed !
     * @endverbatim
     * Further, multiple reads may occur before, during and after
     * a write operation simultaneously. The buffer needs readers+2*writers
     * elements to be guaranteed non blocking.
     * @ingroup PortBuffers
     */
    template<class T>
    class DataObjectLockFree
        : public DataObjectInterface<T>
    {
    public:
        /**
         * The type of the data.
         */
        typedef T DataType;

        /**
         * @brief The maximum number of threads.
         *
         * When used in data flow, this is always 2.
         */
        const unsigned int MAX_THREADS; // = 2
    private:
        /**
         * Conversion of number of threads to size of buffer.
         */
        const unsigned int BUF_LEN; // = MAX_THREADS+2

        /**
         * Internal buffer structure.
         * Both the read and write pointers pointing to this struct
         * must be declared volatile, since they are modified in other threads.
         * I did not declare data as volatile,
         * since we only read/write it in secured buffers.
         */
        struct DataBuf {
            DataBuf()
                : data(), counter(), next()
            {
                oro_atomic_set(&counter, 0);
            }
            DataType data; mutable oro_atomic_t counter; DataBuf* next;
        };

        typedef DataBuf* volatile VolPtrType;
        typedef DataBuf  ValueType;
        typedef DataBuf* PtrType;

        VolPtrType read_ptr;
        VolPtrType write_ptr;

        /**
         * A 3 element Data buffer
         */
        DataBuf* data;
    public:

        /**
         * Construct a DataObjectLockFree by name.
         *
         * @param _name The name of this DataObject.
         * @param initial_value The initial value of this DataObject.
         */
        DataObjectLockFree( const T& initial_value = T(), unsigned int max_threads = 2 )
            : MAX_THREADS(max_threads), BUF_LEN( max_threads + 2),
              read_ptr(0),
              write_ptr(0)
        {
            data = new DataBuf[BUF_LEN];
            read_ptr = &data[0];
            write_ptr = &data[1];
            data_sample(initial_value);
        }

        ~DataObjectLockFree() {
            delete[] data;
        }

        /**
         * Get a copy of the data.
         * This method will allocate memory twice if data is not a value type.
         * Use Get(DataType&) for the non-allocating version.
         *
         * @return A copy of the data.
         */
        virtual DataType Get() const {DataType cache; Get(cache); return cache; }

        /**
         * Get a copy of the Data (non allocating).
         * If pull has reserved enough memory to store the copy,
         * no memory will be allocated.
         *
         * @param pull A copy of the data.
         */
        virtual void Get( DataType& pull ) const
        {
            PtrType reading;
            // loop to combine Read/Modify of counter
            // This avoids a race condition where read_ptr
            // could become write_ptr ( then we would read corrupted data).
            do {
                reading = read_ptr;            // copy buffer location
                oro_atomic_inc(&reading->counter); // lock buffer, no more writes 不能在读的过程中进行写操作
                // XXX smp_mb
                if ( reading != read_ptr )     // if read_ptr changed,
                    oro_atomic_dec(&reading->counter); // better to start over.
                else
                    break;
            } while ( true );

            // from here on we are sure that 'reading'
            // is a valid buffer to read from.
            // 这里是关键，由于用户自定义数据不一定是natively aligned，所以读操作不一定是atomic的，可能在读取自定义数据的过程中被其他线程中断(如再次进行写操作)。 
            pull = reading->data;               // takes some time
            // XXX smp_mb
            oro_atomic_dec(&reading->counter);       // release buffer
        }

        /**
         * Set the data to a certain value (non blocking).
         *
         * @param push The data which must be set.
         */
        virtual void Set( const DataType& push )
        {
            /**
             * This method can not be called concurrently (only one
             * producer). With a minimum of 3 buffers, if the
             * write_ptr+1 field is not occupied, it will remain so
             * because the read_ptr is at write_ptr-1 (and can
             * not increment the counter on write_ptr+1). Hence, no
             * locking is needed.
             */
            // writeout in any case，只有一个producer,并且write_ptr!=read_ptr，所以永远是安全的！
            write_ptr->data = push;
            PtrType wrote_ptr = write_ptr;
            // if next field is occupied (by read_ptr or counter),
            // go to next and check again...
            // 环形buffer，如果是前一个还没有读完，这时候新的写进去又加进来新的读取就会有这种事情发生;
            // 如果前一个读取没有完成(就是那个takes some time的地方)，后续最多能再进行两次写操作（buffer大小为4）.
            while ( oro_atomic_read( &write_ptr
            ->next->counter ) != 0 || write_ptr->next == read_ptr )
                {
                    write_ptr = write_ptr->next;
                    if (write_ptr == wrote_ptr)
                        return; // nothing found, to many readers !
                }

            // we will be able to move, so replace read_ptr
            read_ptr  = wrote_ptr;
            write_ptr = write_ptr->next; // we checked this in the while loop
        }

        virtual void data_sample( const DataType& sample ) {
            // prepare the buffer.
            for (unsigned int i = 0; i < BUF_LEN-1; ++i) {
                data[i].data = sample;
                data[i].next = &data[i+1];
            }
            data[BUF_LEN-1].data = sample;
            data[BUF_LEN-1].next = &data[0];
        }
    };
}}

reference link:
1. https://msdn.microsoft.com/en-us/library/windows/desktop/ee418650(v=vs.85).aspx
2. http://www.boost.org/doc/libs/1_60_0/doc/html/lockfree.html
3. http://preshing.com/20150402/you-can-do-any-kind-of-atomic-read-modify-write-operation/
4. http://preshing.com/20120612/an-introduction-to-lock-free-programming/
5. https://en.wikipedia.org/wiki/ABA_problem