Linux下使用ZMQ实践“生产者-消费者”模型

一、背景

    上一篇文章《Linux下使用ZMQ实践“请求-响应”模型》引入了ZMQ的实践案例，本章继续实践编程模型中常用的“生产者-消费者”模型。

二、相关知识

    ZMQ_PUSH、ZMQ_PULL模型是单发单收的模型，你只能在ZMQ_PUSH套接字上进行send操作，而不能进行recv，反之一样：

       ZMQ_PUSH
           A socket of type ZMQ_PUSH is used by a pipeline node to send messages to
           downstream pipeline nodes. Messages are round-robined to all connected
           downstream nodes. The zmq_recv() function is not implemented for this socket
           type.

           When a ZMQ_PUSH socket enters the mute state due to having reached the high
           water mark for all downstream nodes, or if there are no downstream nodes at
           all, then any zmq_send(3) operations on the socket shall block until the mute
           state ends or at least one downstream node becomes available for sending;
           messages are not discarded.
       ZMQ_PULL
           A socket of type ZMQ_PULL is used by a pipeline node to receive messages from
           upstream pipeline nodes. Messages are fair-queued from among all connected
           upstream nodes. The zmq_send() function is not implemented for this socket
           type.

三、实践

1、“多个生产者-单个消费者”模型

先从main函数入手，当有传参时（./queue 100）走生成者逻辑、无参数时（./queue）走消费者逻辑

int main(int argc, char *argv[])
{
        void *ctx = zmq_ctx_new();
        srandom(time(NULL));

        if (argc > 1) {
                test_producer(ctx, atoi(argv[1]));
        }
        else {
                test_consumer(ctx);
        }
        zmq_ctx_destroy(ctx);
        exit(EXIT_SUCCESS);
}

消费者循环等待数据到来

int test_consumer(void *ctx)
{
        int cnt = 0;
        char id[16] = {0};
        char request[1024];
        void *sock = zmq_socket(ctx, ZMQ_PULL);
        s_set_id_ex(sock, id, sizeof(id));
        zmq_bind(sock, "ipc://queue.ipc");
        LOGN("Consumer %s start\n", id);
        while (++cnt) {
                s_recv(sock, request);
                LOGN("Consumer %s recv: %s\n", id, request);
                usleep(300 * 1000);
        }
        LOGN("Consumer %s stop\n", id);
        zmq_close(sock);
}

生产者则进行消息推送，在消息体中加入来源id、序列号信息

void test_producer(void *ctx, int times)
{
        int ix = 0;
        char id[16] = {0};
        char request[1024];
        void *sock = zmq_socket(ctx, ZMQ_PUSH);
        s_set_id_ex(sock, id, sizeof(id));
        zmq_connect(sock, "ipc://queue.ipc");
        LOGN("Producer %s start\n", id);
        for (ix = 0; ix < times; ix++) {
                snprintf(request, sizeof(request), "Data-%03s-%03d", id, ix);
                s_send(sock, request);
                LOGN("Producer %s send: %s\n", id, request);
                usleep(300 * 1000);
        }
        LOGN("Producer %s stop\n", id);
        zmq_close(sock);
}

运行结果：producer-0027、producer-0082同时给consumer-00C7推送消息，consumer侧进行类似“负载均衡”的消息处理

./queue 10                 
[ 1520654736.979 ]: Producer 0027 start
[ 1520654736.979 ]: Producer 0027 send: Data-0027-000
[ 1520654737.293 ]: Producer 0027 send: Data-0027-001
[ 1520654737.606 ]: Producer 0027 send: Data-0027-002
[ 1520654737.917 ]: Producer 0027 send: Data-0027-003
[ 1520654738.222 ]: Producer 0027 send: Data-0027-004
[ 1520654738.535 ]: Producer 0027 send: Data-0027-005
[ 1520654738.848 ]: Producer 0027 send: Data-0027-006
[ 1520654739.158 ]: Producer 0027 send: Data-0027-007
[ 1520654739.470 ]: Producer 0027 send: Data-0027-008
[ 1520654739.775 ]: Producer 0027 send: Data-0027-009
[ 1520654740.088 ]: Producer 0027 stop

./queue 10                 
[ 1520654737.390 ]: Producer 0082 start
[ 1520654737.390 ]: Producer 0082 send: Data-0082-000
[ 1520654737.698 ]: Producer 0082 send: Data-0082-001
[ 1520654738.011 ]: Producer 0082 send: Data-0082-002
[ 1520654738.316 ]: Producer 0082 send: Data-0082-003
[ 1520654738.629 ]: Producer 0082 send: Data-0082-004
[ 1520654738.941 ]: Producer 0082 send: Data-0082-005
[ 1520654739.251 ]: Producer 0082 send: Data-0082-006
[ 1520654739.564 ]: Producer 0082 send: Data-0082-007
[ 1520654739.869 ]: Producer 0082 send: Data-0082-008
[ 1520654740.174 ]: Producer 0082 send: Data-0082-009
[ 1520654740.487 ]: Producer 0082 stop

./queue                  
[ 1520654721.190 ]: Consumer 00C7 start
[ 1520654736.980 ]: Consumer 00C7 recv: Data-0027-000
[ 1520654737.294 ]: Consumer 00C7 recv: Data-0027-001
[ 1520654737.606 ]: Consumer 00C7 recv: Data-0082-000
[ 1520654737.917 ]: Consumer 00C7 recv: Data-0082-001
[ 1520654738.222 ]: Consumer 00C7 recv: Data-0082-002
[ 1520654738.535 ]: Consumer 00C7 recv: Data-0082-003
[ 1520654738.848 ]: Consumer 00C7 recv: Data-0082-004
[ 1520654739.158 ]: Consumer 00C7 recv: Data-0082-005
[ 1520654739.470 ]: Consumer 00C7 recv: Data-0082-006
[ 1520654739.775 ]: Consumer 00C7 recv: Data-0082-007
[ 1520654740.088 ]: Consumer 00C7 recv: Data-0082-008
[ 1520654740.393 ]: Consumer 00C7 recv: Data-0082-009
[ 1520654740.697 ]: Consumer 00C7 recv: Data-0027-002
[ 1520654741.010 ]: Consumer 00C7 recv: Data-0027-003
[ 1520654741.315 ]: Consumer 00C7 recv: Data-0027-004
[ 1520654741.615 ]: Consumer 00C7 recv: Data-0027-005
[ 1520654741.921 ]: Consumer 00C7 recv: Data-0027-006
[ 1520654742.226 ]: Consumer 00C7 recv: Data-0027-007
[ 1520654742.538 ]: Consumer 00C7 recv: Data-0027-008
[ 1520654742.851 ]: Consumer 00C7 recv: Data-0027-009

由于使用多生产者、单消费者，所以在生产者中使用 zmq_connect() 方法，消费者中使用 zmq_bind() 方法；

（多个进程无法对同一个地址进行绑定）

2、“单生产者-多消费者”模型

代码只要进行微小改动：生产者中使用 zmq_bind() 方法，消费者中使用 zmq_connect() 方法；

运行结果如下：producer-0084推出10个消息、分别被consumer-0003、consumer-00CA接收处理

./queue 10
[ 1520656647.700 ]: Producer 0084 start
[ 1520656647.848 ]: Producer 0084 send: Data-0084-000
[ 1520656648.152 ]: Producer 0084 send: Data-0084-001
[ 1520656648.464 ]: Producer 0084 send: Data-0084-002
[ 1520656648.777 ]: Producer 0084 send: Data-0084-003
[ 1520656649.089 ]: Producer 0084 send: Data-0084-004
[ 1520656649.396 ]: Producer 0084 send: Data-0084-005
[ 1520656649.709 ]: Producer 0084 send: Data-0084-006
[ 1520656650.021 ]: Producer 0084 send: Data-0084-007
[ 1520656650.324 ]: Producer 0084 send: Data-0084-008
[ 1520656650.637 ]: Producer 0084 send: Data-0084-009
[ 1520656650.950 ]: Producer 0084 stop

./queue 
[ 1520656640.402 ]: Consumer 0003 start
[ 1520656648.464 ]: Consumer 0003 recv: Data-0084-002
[ 1520656649.089 ]: Consumer 0003 recv: Data-0084-004
[ 1520656649.709 ]: Consumer 0003 recv: Data-0084-006
[ 1520656650.324 ]: Consumer 0003 recv: Data-0084-008

./queue 
[ 1520656638.389 ]: Consumer 00CA start
[ 1520656647.848 ]: Consumer 00CA recv: Data-0084-000
[ 1520656648.152 ]: Consumer 00CA recv: Data-0084-001
[ 1520656648.777 ]: Consumer 00CA recv: Data-0084-003
[ 1520656649.396 ]: Consumer 00CA recv: Data-0084-005
[ 1520656650.021 ]: Consumer 00CA recv: Data-0084-007
[ 1520656650.637 ]: Consumer 00CA recv: Data-0084-009

四、总结

    使用ZMQ_PUSH、ZMQ_PULL队列模式可以方便地实现“单生产者-单消费者”、“多生产者-单消费者”、“单生产者-多消费者”模型，支持多线程、多进程的处理，而且在应用层可以完全不用考虑队列加锁的问题（ZMQ内部处理），开发起来效率更快；

    ZMQ的编程哲学就是消息传递来避免锁：我们需要并行地做一个计算处理，不是直接去调用计算单元的接口（这样会涉及到多线程竞争的问题），而是通过传递消息的形式（请求-响应、负载均衡、远程过程调用）给计算子单元去处理；

    看到这里，同时也引发了一个问题：“多生产者-多消费者”应该如何设计呢？这个场景在分布式处理用的就比较多了，而且实现难度更大（如何保证消息来源去向一致、如何保证消息可靠性），这也就不仅仅是PUSH、PULL能解决了，ZMQ还有其他套件呢，让我们拭目以待！

参考文章：

[1] http://zguide.zeromq.org/page:all