Linux下使用ZMQ实践“生产者-消费者”模型

linux 同时被 2 个专栏收录
52 篇文章 2 订阅
13 篇文章 0 订阅

一、背景

    上一篇文章《Linux下使用ZMQ实践“请求-响应”模型》引入了ZMQ的实践案例,本章继续实践编程模型中常用的“生产者-消费者”模型。

二、相关知识

    ZMQ_PUSH、ZMQ_PULL模型是单发单收的模型,你只能在ZMQ_PUSH套接字上进行send操作,而不能进行recv,反之一样:

       ZMQ_PUSH
           A socket of type ZMQ_PUSH is used by a pipeline node to send messages to
           downstream pipeline nodes. Messages are round-robined to all connected
           downstream nodes. The zmq_recv() function is not implemented for this socket
           type.

           When a ZMQ_PUSH socket enters the mute state due to having reached the high
           water mark for all downstream nodes, or if there are no downstream nodes at
           all, then any zmq_send(3) operations on the socket shall block until the mute
           state ends or at least one downstream node becomes available for sending;
           messages are not discarded.
       ZMQ_PULL
           A socket of type ZMQ_PULL is used by a pipeline node to receive messages from
           upstream pipeline nodes. Messages are fair-queued from among all connected
           upstream nodes. The zmq_send() function is not implemented for this socket
           type.

三、实践

1、“多个生产者-单个消费者”模型


先从main函数入手,当有传参时(./queue 100)走生成者逻辑、无参数时(./queue)走消费者逻辑

int main(int argc, char *argv[])
{
        void *ctx = zmq_ctx_new();
        srandom(time(NULL));

        if (argc > 1) {
                test_producer(ctx, atoi(argv[1]));
        }
        else {
                test_consumer(ctx);
        }
        zmq_ctx_destroy(ctx);
        exit(EXIT_SUCCESS);
}

消费者循环等待数据到来

int test_consumer(void *ctx)
{
        int cnt = 0;
        char id[16] = {0};
        char request[1024];
        void *sock = zmq_socket(ctx, ZMQ_PULL);
        s_set_id_ex(sock, id, sizeof(id));
        zmq_bind(sock, "ipc://queue.ipc");
        LOGN("Consumer %s start\n", id);
        while (++cnt) {
                s_recv(sock, request);
                LOGN("Consumer %s recv: %s\n", id, request);
                usleep(300 * 1000);
        }
        LOGN("Consumer %s stop\n", id);
        zmq_close(sock);
}

生产者则进行消息推送,在消息体中加入来源id、序列号信息

void test_producer(void *ctx, int times)
{
        int ix = 0;
        char id[16] = {0};
        char request[1024];
        void *sock = zmq_socket(ctx, ZMQ_PUSH);
        s_set_id_ex(sock, id, sizeof(id));
        zmq_connect(sock, "ipc://queue.ipc");
        LOGN("Producer %s start\n", id);
        for (ix = 0; ix < times; ix++) {
                snprintf(request, sizeof(request), "Data-%03s-%03d", id, ix);
                s_send(sock, request);
                LOGN("Producer %s send: %s\n", id, request);
                usleep(300 * 1000);
        }
        LOGN("Producer %s stop\n", id);
        zmq_close(sock);
}

运行结果:producer-0027、producer-0082同时给consumer-00C7推送消息,consumer侧进行类似“负载均衡”的消息处理

./queue 10                 
[ 1520654736.979 ]: Producer 0027 start
[ 1520654736.979 ]: Producer 0027 send: Data-0027-000
[ 1520654737.293 ]: Producer 0027 send: Data-0027-001
[ 1520654737.606 ]: Producer 0027 send: Data-0027-002
[ 1520654737.917 ]: Producer 0027 send: Data-0027-003
[ 1520654738.222 ]: Producer 0027 send: Data-0027-004
[ 1520654738.535 ]: Producer 0027 send: Data-0027-005
[ 1520654738.848 ]: Producer 0027 send: Data-0027-006
[ 1520654739.158 ]: Producer 0027 send: Data-0027-007
[ 1520654739.470 ]: Producer 0027 send: Data-0027-008
[ 1520654739.775 ]: Producer 0027 send: Data-0027-009
[ 1520654740.088 ]: Producer 0027 stop
./queue 10                 
[ 1520654737.390 ]: Producer 0082 start
[ 1520654737.390 ]: Producer 0082 send: Data-0082-000
[ 1520654737.698 ]: Producer 0082 send: Data-0082-001
[ 1520654738.011 ]: Producer 0082 send: Data-0082-002
[ 1520654738.316 ]: Producer 0082 send: Data-0082-003
[ 1520654738.629 ]: Producer 0082 send: Data-0082-004
[ 1520654738.941 ]: Producer 0082 send: Data-0082-005
[ 1520654739.251 ]: Producer 0082 send: Data-0082-006
[ 1520654739.564 ]: Producer 0082 send: Data-0082-007
[ 1520654739.869 ]: Producer 0082 send: Data-0082-008
[ 1520654740.174 ]: Producer 0082 send: Data-0082-009
[ 1520654740.487 ]: Producer 0082 stop
./queue                  
[ 1520654721.190 ]: Consumer 00C7 start
[ 1520654736.980 ]: Consumer 00C7 recv: Data-0027-000
[ 1520654737.294 ]: Consumer 00C7 recv: Data-0027-001
[ 1520654737.606 ]: Consumer 00C7 recv: Data-0082-000
[ 1520654737.917 ]: Consumer 00C7 recv: Data-0082-001
[ 1520654738.222 ]: Consumer 00C7 recv: Data-0082-002
[ 1520654738.535 ]: Consumer 00C7 recv: Data-0082-003
[ 1520654738.848 ]: Consumer 00C7 recv: Data-0082-004
[ 1520654739.158 ]: Consumer 00C7 recv: Data-0082-005
[ 1520654739.470 ]: Consumer 00C7 recv: Data-0082-006
[ 1520654739.775 ]: Consumer 00C7 recv: Data-0082-007
[ 1520654740.088 ]: Consumer 00C7 recv: Data-0082-008
[ 1520654740.393 ]: Consumer 00C7 recv: Data-0082-009
[ 1520654740.697 ]: Consumer 00C7 recv: Data-0027-002
[ 1520654741.010 ]: Consumer 00C7 recv: Data-0027-003
[ 1520654741.315 ]: Consumer 00C7 recv: Data-0027-004
[ 1520654741.615 ]: Consumer 00C7 recv: Data-0027-005
[ 1520654741.921 ]: Consumer 00C7 recv: Data-0027-006
[ 1520654742.226 ]: Consumer 00C7 recv: Data-0027-007
[ 1520654742.538 ]: Consumer 00C7 recv: Data-0027-008
[ 1520654742.851 ]: Consumer 00C7 recv: Data-0027-009

由于使用多生产者、单消费者,所以在生产者中使用 zmq_connect() 方法,消费者中使用 zmq_bind() 方法;

(多个进程无法对同一个地址进行绑定)

2、“单生产者-多消费者”模型

代码只要进行微小改动:生产者中使用 zmq_bind() 方法,消费者中使用 zmq_connect() 方法;

运行结果如下:producer-0084推出10个消息、分别被consumer-0003、consumer-00CA接收处理

./queue 10
[ 1520656647.700 ]: Producer 0084 start
[ 1520656647.848 ]: Producer 0084 send: Data-0084-000
[ 1520656648.152 ]: Producer 0084 send: Data-0084-001
[ 1520656648.464 ]: Producer 0084 send: Data-0084-002
[ 1520656648.777 ]: Producer 0084 send: Data-0084-003
[ 1520656649.089 ]: Producer 0084 send: Data-0084-004
[ 1520656649.396 ]: Producer 0084 send: Data-0084-005
[ 1520656649.709 ]: Producer 0084 send: Data-0084-006
[ 1520656650.021 ]: Producer 0084 send: Data-0084-007
[ 1520656650.324 ]: Producer 0084 send: Data-0084-008
[ 1520656650.637 ]: Producer 0084 send: Data-0084-009
[ 1520656650.950 ]: Producer 0084 stop
./queue 
[ 1520656640.402 ]: Consumer 0003 start
[ 1520656648.464 ]: Consumer 0003 recv: Data-0084-002
[ 1520656649.089 ]: Consumer 0003 recv: Data-0084-004
[ 1520656649.709 ]: Consumer 0003 recv: Data-0084-006
[ 1520656650.324 ]: Consumer 0003 recv: Data-0084-008
./queue 
[ 1520656638.389 ]: Consumer 00CA start
[ 1520656647.848 ]: Consumer 00CA recv: Data-0084-000
[ 1520656648.152 ]: Consumer 00CA recv: Data-0084-001
[ 1520656648.777 ]: Consumer 00CA recv: Data-0084-003
[ 1520656649.396 ]: Consumer 00CA recv: Data-0084-005
[ 1520656650.021 ]: Consumer 00CA recv: Data-0084-007
[ 1520656650.637 ]: Consumer 00CA recv: Data-0084-009

四、总结

    使用ZMQ_PUSH、ZMQ_PULL队列模式可以方便地实现“单生产者-单消费者”、“多生产者-单消费者”、“单生产者-多消费者”模型,支持多线程、多进程的处理,而且在应用层可以完全不用考虑队列加锁的问题(ZMQ内部处理),开发起来效率更快;

    ZMQ的编程哲学就是消息传递来避免锁:我们需要并行地做一个计算处理,不是直接去调用计算单元的接口(这样会涉及到多线程竞争的问题),而是通过传递消息的形式(请求-响应、负载均衡、远程过程调用)给计算子单元去处理;

    看到这里,同时也引发了一个问题:“多生产者-多消费者”应该如何设计呢?这个场景在分布式处理用的就比较多了,而且实现难度更大(如何保证消息来源去向一致、如何保证消息可靠性),这也就不仅仅是PUSH、PULL能解决了,ZMQ还有其他套件呢,让我们拭目以待!


参考文章:

[1] http://zguide.zeromq.org/page:all

  • 0
    点赞
  • 0
    评论
  • 1
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

表情包
插入表情
评论将由博主筛选后显示,对所有人可见 | 还能输入1000个字符
©️2021 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值