ovs 有限队列 mpsc 实现分析

插入实现

插入实现原子插入，即便进入非稳态，也支持多个生产者并发插入，且有序：

/* Multi-producer, single-consumer queue
 * =====================================
 *
 * This an implementation of the MPSC queue described by Dmitri Vyukov [1].
 *
 * One atomic exchange operation is done per insertion.  Removal in most cases
 * will not require atomic operation and will use one atomic exchange to close
 * the queue chain.
 *
 * Insertion
 * =========
 *
 * The queue is implemented using a linked-list.  Insertion is done at the
 * back of the queue, by swapping the current end with the new node atomically,
 * then pointing the previous end toward the new node.  To follow Vyukov
 * nomenclature, the end-node of the chain is called head.  A producer will
 * only manipulate the head.
 *
 * The head swap is atomic, however the link from the previous head to the new
 * one is done in a separate operation.  This means that the chain is
 * momentarily broken, when the previous head still points to NULL and the
 * current head has been inserted.
 *
 * Considering a series of insertions, the queue state will remain consistent
 * and the insertions order is compatible with their precedence, thus the
 * queue is serializable.  However, because an insertion consists in two
 * separate memory transactions, it is not linearizable.
 */
 
 void
mpsc_queue_insert(struct mpsc_queue *queue, struct mpsc_queue_node *node)
{
    struct mpsc_queue_node *prev;

    //首先初始化好尾节点next指针
    atomic_store_relaxed(&node->next, NULL);
    //原子替换生产者ProducerIndex指针head，指向新的node节点，并返回原来的head节点
    prev = atomic_exchange_explicit(&queue->head, node, memory_order_acq_rel);
    //更新原来的生产者head节点next，指向新的node
    atomic_store_explicit(&prev->next, node, memory_order_release);
}

删除实现

 /* Removal
 * =======
 *
 * The consumer must deal with the queue inconsistency.  It will manipulate
 * the tail of the queue and move it along the latest consumed elements.
 * When an end of the chain of elements is found (the next pointer is NULL),
 * the tail is compared with the head.
 *
 * If both points to different addresses, then the queue is in an inconsistent
 * state: the tail cannot move forward as the next is NULL, but the head is not
 * the last element in the chain: this can only happen if the chain is broken.
 *
 * In this case, the consumer must wait for the producer to finish writing the
 * next pointer of its current tail: 'MPSC_QUEUE_RETRY' is returned.
 *
 * Removal is thus in most cases (when there are elements in the queue)
 * accomplished without using atomics, until the last element of the queue.
 * There, the head is atomically loaded. If the queue is in a consistent state,
 * the head is moved back to the queue stub by inserting the stub in the queue:
 * ending the queue is the same as an insertion, which is one atomic XCHG.
 */
struct mpsc_queue_node *
mpsc_queue_pop(struct mpsc_queue *queue)
    OVS_REQUIRES(queue->read_lock)
{
    enum mpsc_queue_poll_result result;
    struct mpsc_queue_node *node;

    do {
        result = mpsc_queue_poll(queue, &node);
        if (result == MPSC_QUEUE_EMPTY) {
            return NULL;
        }
    //如果当前正好有生产者插入，且链表处于非稳态，则等待重试
    } while (result == MPSC_QUEUE_RETRY);

    return node;
}

enum mpsc_queue_poll_result {
    /* Queue is empty. */
    MPSC_QUEUE_EMPTY,
    /* Polling the queue returned an item. */
    MPSC_QUEUE_ITEM,
    /* Data has been enqueued but one or more producer thread have not
     * finished writing it. The queue is in an inconsistent state.
     * Retrying shortly, if the producer threads are still active, will
     * return the data.
     */
    MPSC_QUEUE_RETRY,
};

enum mpsc_queue_poll_result
mpsc_queue_poll(struct mpsc_queue *queue, struct mpsc_queue_node **node)
    OVS_REQUIRES(queue->read_lock)
{
    struct mpsc_queue_node *tail;
    struct mpsc_queue_node *next;
    struct mpsc_queue_node *head;

    atomic_read_relaxed(&queue->tail, &tail);
    atomic_read_explicit(&tail->next, &next, memory_order_acquire);

    //消费者指针指向链表头部，比如此时消费者第一次开始消费
    if (tail == &queue->stub) {
        if (next == NULL) {
            return MPSC_QUEUE_EMPTY;
        }
        //消费者第一次消费，且队列中已存在待消费元素，移动tail指针，跳过头节点
        atomic_store_relaxed(&queue->tail, next);
        tail = next;
        atomic_read_explicit(&tail->next, &next, memory_order_acquire);
    }

    if (next != NULL) {
        //大多数情况下，如果当前消费的不是队列中最后一个元素，则可以直接返回queue->tail 指针指向的节点了
        atomic_store_relaxed(&queue->tail, next);
        *node = tail;
        return MPSC_QUEUE_ITEM;
    }
    //少数情况下，如果当前next指针为null，则说明当前即将返回的节点，是队列的最后一个节点。此时需要
    //额外的操作，除了返回最后一个节点外，还要将head、tail指针指向队列头部
    atomic_read_explicit(&queue->head, &head, memory_order_acquire);
    if (tail != head) {
        //如果tail->next为空，但是此时tail和head 却不相等（指向不同的链表节点），说明当前链表处于非稳态，
        //有生产者正在插入。所以我们稍后重试
        return MPSC_QUEUE_RETRY;
    }

当链表仅剩最后一个节点时，事情会变得复杂一点，此时会将head 重新指回队列头处。这是通过将stub插入链表实现的：

    mpsc_queue_insert(queue, &queue->stub);

    atomic_read_explicit(&tail->next, &next, memory_order_acquire);
    if (next != NULL) {
        atomic_store_relaxed(&queue->tail, next);
        *node = tail;
        return MPSC_QUEUE_ITEM;
    }

上面我们假设的是：

1. next=null 后，检查head 和tail 指向不同地方，说明当前非稳态，所以返回。
2. 以及head 和tail 指向相同地方，说明都指向最后一个节点，所以重置head 和tail 指针并返回最后一个节点

这里还有第三种情况：

next=null 且 head 和tail指向同一个节点，但在执行前述代码时，同时有生产者插入新节点，与消费者插入stub节点形成并发。如下图。

前面虽然检测了非稳态，tail 和head 不相等的情况并return。但是后面的所有代码，仍然可能存在head 和tail不相等，因为随时生产者都可能插入。只要出现并发插入（一个消费者插入stub、一个生产者插入新节点，且生产者没有完成插入，导致链表为非稳态，即生产者prev指针还没有更新为指向新节点），则此时tail->next=null，如下图所示：

所以，最终我们的处理是，只要并发的生产者进入稳态，我们就能返回tail节点作为pop结果。而如果此时并发的生产者为非稳态，则我们就返回empty，即便此时链表中甚至可能存在两个节点可消费，如下图状态c：

    return MPSC_QUEUE_EMPTY;
}

参考：

1. https://blog.csdn.net/qq_32099833/article/details/117547853