zeromq一直困扰我的问题和自我反省

今天去zeromq的maillist发信问了两个一直困扰我几天的问题了，结果迅速收到回复，立即想明白了，大牛还是很多的，看来以后自己要把代码再吃得透点，不明白的地方及时问大牛，能节省很多时间:

1.  zeromq的多个I/O线程之间没有使用同步机制，会不会产生问题?

我本来以为需要加入类似于leader/followers的机制来保证消息的接收和发送不会有问题，因此我一直在绕死了这个地方。结果得到了回答之后，原来是因为在zeromq中多个I/O线程间不会共享同一个socket句柄。

Q:

Hi all:

    I read the source code of zeromq, but I don't know how to guarantee
message sending/recving sync when using multi io thread. For example, if
one thread recv a part of message, and then another thread recv this
message and can not process this message correctly. so I think it should

use a sync facility likes 'leader/followers' to prevent this problem.

Kaka Chen

A:

Hi Kaka,

Multiple threads can't share a socket, so you shouldn't run into a problem here. If you receive a part, you can check the ZMQ_RCVMORE sockopt flag to see whether there are any more parts - you are guaranteed to get an entire message without parts from other messages being interleaved until that flag returns 0.

Ian

2. zeromq中如何重复使用消息？

因为我看了sample和test中都是在循环中不断的调用zmq_init_msg_size()重新初始化消息，因此消息大小为1M循环10000次我的机器就报内存分配错误了。

于是我就在想是否能在循环外建立和释放消息，而在循环中反复使用这条消息发送呢，结果我尝试了发现不能。后来我去看代码实现，终于明白了一些:

原来zeromq在消息写入到outbound的管道中时就把消息所有权交给了管道了，而自己变成空消息返回给上层，而管道管理着消息的生命周期。这样一想，就觉得应该要改代码采用引用计数的方式去重新实现zeromq的消息机制，于是我问了这个问题。

 结果经过大牛Chuck Remes的鄙视之后我终于明白了:

原来当使用zmq_msg_data_init()的时候设置free函数为NULL就不会在close的时候销毁data了，因此zmq_msg_data_init()给我们提供了自己管理消息data的内存的方式。

而zmq_msg_init_size()则是会将data释放，因为data是在content结构体之后的size部分的heap上。

大牛还提供了另外一种方式，就是使用zmq_copy_data()来增加引用计数保留消息不被free。

Q:

Hi All:

    From source code of zeromq, I found it writes message to outbound pipe, the ownership is transfered to pipe, and the original message was re-initialized to a empty size message by zmq_msg_init(1). This strategy prevent me to reuse a message for test because I can not manage this message's life-cycle and the pipe will manage it. So I can not test 1M size message for looping 10000 times, it will occur mem allocate problems. Is it has some method to reuse a message in zeromq? what about try to use ref-count when writing message
to outbound pipe by override operator = () function in message_size_t.

Kaka Chen

A:

I can tell you haven't read the man pages or the guide yet. There are two ways of accomplishing your goal.

One, take a look at zmq_msg_init_data() [1]. It creates a zmq_msg_t structure and uses the buffer that you pass to it. It also takes an *optional* function pointer to deallocate the buffer when 0mq is done with it. If you leave that as NULL, then 0mq will not deallocate the buffer. Therefore, you could call zmq_msg_init_data() in a tight loop and pass it the same buffer over and over again (zero-copy).

Alternately, you could also use zmq_msg_copy() [2] to copy the data buffer from a "src" message to a new "dst" message. The underlying implementation does *not* do a memcpy(); instead, it increments a reference count. A later call to zmq_msg_close() decrements that count. When the count reaches 0, the deallocation function is called to release the memory.

I highly recommend that you read through the documentation. Your questions are all answered in the guide or in the man pages.

cr

[1] http://api.zeromq.org/2-1:zmq-msg-init-data

[2] http://api.zeromq.org/2-0:zmq-msg-copy
______________________________

_________________
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

总结:

1. maillist是好东西。

2. 以后读代码要更加认真仔细。

3. 要好好看看guide或者man pages。

4. 明天我要好好做下实验，试试看这两种方案。

下面贴出两种方案的代码，是根据perf/remote_thr.cpp改的:

/*
 * Test throughput with shared data
*/

#include "../include/zmq.h"
#include "../include/zmq_utils.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main (int argc, char *argv [])
{
    const char *connect_to;
    int message_count;
    int message_size;
    void *ctx;
    void *s;
    int rc;
    int i;
    zmq_msg_t msg;

    if (argc != 4) {
        printf ("usage: remote_thr <connect-to> <message-size> "
            "<message-count>\n");
        return 1;
    }
    connect_to = argv [1];
    message_size = atoi (argv [2]);
    message_count = atoi (argv [3]);

    ctx = zmq_init (1);
    if (!ctx) {
        printf ("error in zmq_init: %s\n", zmq_strerror (errno));
        return -1;
    }

    s = zmq_socket (ctx, ZMQ_PUB);
    if (!s) {
        printf ("error in zmq_socket: %s\n", zmq_strerror (errno));
        return -1;
    }

    //  Add your socket options here.
    //  For example ZMQ_RATE, ZMQ_RECOVERY_IVL and ZMQ_MCAST_LOOP for PGM.

    rc = zmq_connect (s, connect_to);
    if (rc != 0) {
        printf ("error in zmq_connect: %s\n", zmq_strerror (errno));
        return -1;
    }

    char *data = (char *)malloc (sizeof (char) * message_size);

    for (i = 0; i != message_count; i++) {
        rc = zmq_msg_init_data (&msg, data, message_size, NULL, NULL);
        if (rc != 0) {
            printf ("error in zmq_msg_init_data: %s\n", zmq_strerror (errno));
            return -1;
        }
#if defined ZMQ_MAKE_VALGRIND_HAPPY
        memset (zmq_msg_data (&msg), 0, message_size);
#endif
        rc = zmq_send (s, &msg, 0);
        if (rc != 0) {
            printf ("error in zmq_send: %s\n", zmq_strerror (errno));
            return -1;
        }
        rc = zmq_msg_close (&msg);
        if (rc != 0) {
            printf ("error in zmq_msg_close: %s\n", zmq_strerror (errno));
            return -1;
        }
    }
    

    rc = zmq_close (s);
    if (rc != 0) {
        printf ("error in zmq_close: %s\n", zmq_strerror (errno));
        return -1;
    }

    rc = zmq_term (ctx);
    if (rc != 0) {
        printf ("error in zmq_term: %s\n", zmq_strerror (errno));
        return -1;
    }

    if (data) {
        free(data);
    }
    return 0;
}

/*
 * Test throughput with shared data with zmq_msg_copy
*/

#include "../include/zmq.h"
#include "../include/zmq_utils.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main (int argc, char *argv [])
{
    const char *connect_to;
    int message_count;
    int message_size;
    void *ctx;
    void *s;
    int rc;
    int i;
    zmq_msg_t msg;

    if (argc != 4) {
        printf ("usage: remote_thr <connect-to> <message-size> "
            "<message-count>\n");
        return 1;
    }
    connect_to = argv [1];
    message_size = atoi (argv [2]);
    message_count = atoi (argv [3]);

    ctx = zmq_init (1);
    if (!ctx) {
        printf ("error in zmq_init: %s\n", zmq_strerror (errno));
        return -1;
    }

    s = zmq_socket (ctx, ZMQ_PUB);
    if (!s) {
        printf ("error in zmq_socket: %s\n", zmq_strerror (errno));
        return -1;
    }

    //  Add your socket options here.
    //  For example ZMQ_RATE, ZMQ_RECOVERY_IVL and ZMQ_MCAST_LOOP for PGM.

    rc = zmq_connect (s, connect_to);
    if (rc != 0) {
        printf ("error in zmq_connect: %s\n", zmq_strerror (errno));
        return -1;
    }

    zmq_msg_init_size (&msg, message_size);
    for (i = 0; i != message_count; i++) {

        zmq_msg_t msg_copied;
        zmq_msg_init(&msg_copied);
        rc = zmq_msg_copy (&msg_copied, &msg);
        if (rc != 0) {
            printf ("error in zmq_msg_copy: %s\n", zmq_strerror (errno));
            return -1;
        }
#if defined ZMQ_MAKE_VALGRIND_HAPPY
        memset (zmq_msg_data (&msg_copied), 0, message_size);
#endif

        rc = zmq_send (s, &msg_copied, 0);
        if (rc != 0) {
            printf ("error in zmq_send: %s\n", zmq_strerror (errno));
            return -1;
        }
        rc = zmq_msg_close (&msg_copied);
        if (rc != 0) {
            printf ("error in zmq_msg_close: %s\n", zmq_strerror (errno));
            return -1;
        }
    }

    rc = zmq_close (s);
    if (rc != 0) {
        printf ("error in zmq_close: %s\n", zmq_strerror (errno));
        return -1;
    }

    rc = zmq_term (ctx);
    if (rc != 0) {
        printf ("error in zmq_term: %s\n", zmq_strerror (errno));
        return -1;
    }

    return 0;
}