ZeroMQ Guide Chapter 2 Notes

Sockets and Patterns

The Socket API

Similar to BSD sockets:

zmq_socket(), zmq_close()
zmq_setsockopt(), zmq_getsockopt()
zmq_bind(), zmq_connnect()
zmq_msg_send(), zmq_msg_recv()

Sockets are void pointers and messages are structures.

“All sockets belong to ZMQ.”

Plugging Sockets into the Topology

Differences from classic TCP:

• Arbitrary transports
• One socket multiple connections. Thus one socket can be bind to multiple endpoints (protocol + address).
• NO “zmq_accept()” method, once binded start to accept.

About bindand connect:

bindfor “servers”, that is more static parts in the topology network, and connectfor “clients”, that is more dynamic parts.

The “server” does not need to start listening earlier than when the “client” calls connect. The connect request will block in background instead of returning failure at once.

A socket can NOT be binded to the same endpoint twice.

The ipc transport does, however, let one process bind to an endpoint already used by a first process. It’s meant to allow a process to recover after a crash.

Sending and Receiving Messages

Differences from classic TCP:

• ZMQ carries messages instead of byte streams.
• ZMQ process is length-specified.
• ZMQ sockets do I/O in a background thread.
• ZMQ supports 1-to-N routing.

zmq_sendputs the message into background queue for asynchronous sending, and possibly returns with the message still unsent.

Unicast Transports

• (ZMQ’s) tcp :
  • Disconnected: Do not require the other endpoint to be alive when connectto it.
  • Most common used.
• ipc (inter process communication):
  • Disconnected.
  • Not available on Windows yet.
  • Mind the permissions for the processes.
• inproc (inter thread communication):
  • Connected.

I/O Threads

When modifying the number of I/O threads, do call zmq_ctx_set()before creating any sockets.

Messaging Patterns

Built-in core patterns:

• Request-Reply
• Pub-sub
• Pipeline
• Exclusive Pair: connecting 2 threads in a process exclusively.

Valid connect-bind pairs:

• PUB and SUB
• REQ and REP
• REQ and ROUTER (take care, REQ inserts an extra null frame)
• DEALER and REP (take care, REP assumes a null frame)
• DEALER and ROUTER
• DEALER and DEALER
• ROUTER and ROUTER
• PUSH and PULL
• PAIR and PAIR

Working with Messages

Simple APIs: zmq_send, zmq_recv. The latter truncates messages to whatever buffer size you provide.

zmq_msg_tAPIs:

• Initialise a message: zmq_msg_init(), zmq_msg_init_size(), zmq_msg_init_data().
• Sending and receiving a message: zmq_msg_send(), zmq_msg_recv().
• Release a message: zmq_msg_close().
• Access message content: zmq_msg_data(), zmq_msg_size(), zmq_msg_more().
• Work with message properties: zmq_msg_get(), zmq_msg_set().
• Message manipulation: zmq_msg_copy(), zmq_msg_move().

Note: zmq_msg_send()set the message size back to 0, and the message sent can not be accessed again. To achieve that, create another message and call zmq_msg_init, zmq_msg_copy, which will copy a reference to the original one. A message is destroyed only if all copies are closed or sent.

Frame: message parts, also length-specified data block. Each frame is a zmq_msg_t. A “more” flag is used to notify latter frames in a message.

Handling Multiple Sockets

Use zmq_poll()to deal with multiple sockets in different patterns.

Multi-part Messages

• Frames are separately sent and received by calling zmq_msg_send, zmq_msg_recv.
• All frames will be sent altogether.
• You will receive all frames or none frames. In zmq_poll, when the first frame is received, all the rest are arrived.

The Dynamic Discovery Problem

Multiple publisher and multiple subscriber?

Simple Pub-Sub: make a number of $m\times n$ connections.

$\to$ Pub-Sub with a Proxy

$\to$ Extended Pub-Sub: Add XSUB and XPUB sockets to the proxy. All PUBs connects to XSUB and all SUBs connects to XPUB.

The proxy forwards subscription messages from subscriber side to publisher side, by reading them from the XPUB socket and writing them to the XSUB socket.

Shared Queue (DEALER and ROUTER sockets)

How to build a broker on multiple client multiple service pattern just like the Pub-Sub proxy? REQ-REP is blocking and thus not available in this case.

$\to$ Extended Request-Reply: Use ROUTER for the client side (frontend) and DEALER for the service side (backend) on the broker.

This rrbroker makes up the zmq_proxy.

Error Handling

Always handle errors in time in C/C++. eg. NULL, -1, errno by zmq_errno, zmq_strerror.

When one thread calls zmq_ctx_destroy(), and other threads are still doing blocking work, the zmq_ctx_destroy() call closes the context and all blocking calls exit with -1, and errno set to ETERM.

How does SINK send kill messages to the WORKERs when all tasks are finished?

$\to$ Add another PUB-SUB connection. PUB on the SINK and SUBs on all WORKERs.

Interrupt Handling

Detecting Memory Leaks

A detector called valgrind.

Multithreading with ZeroMQ

“Perfect MT programs”: no mutexes, locks, or any other form of inter-thread communication except messages sent across ZeroMQ sockets.

Idea: no share state.

Rules:

• Never share data between threads except ZMQ contexts

• No mutexes, critical sections, semaphores.

• Create CONTEXT at the beginning and pass it to the threads.

• Use attached threads for inproc: bind parent socket, create child threads and connect them to the socket.

  Use detached threads to simulate independent tasks, with their own contexts.

• Don’t share ZeroMQ sockets between threads. ZeroMQ sockets are not threadsafe.

  Do not use or close sockets except in the thread that created them.

Multithreaded server pattern: REQ-ROUTER-queue-DEALER-REP.

Signaling Between Threads (PAIR Sockets)

Relay race example:

Step 1

void step1(&CONTEXT) {
    SOCKET(PAIR);
    CONNECT(addr1);
    SEND;
    CLOSE;
}

Step 2

void step2(&CONTEXT) {
    receiver = SOCKET(PAIR);
    receiver.BIND(addr1);
    THREAD(step1, CONTEXT);
    receiver.RECV();
    receiver.CLOSE();

    xmitter = SOCKET(PAIR);
    xmitter.CONNECT(addr2);
    xmitter.SEND();
    xmitter.CLOSE();
}

Step 3

int main() {
    CONTEXT();

    receiver = SOCKET(PAIR);
    receiver.BIND(addr2);
    THREAD(step2, CONTEXT);
    receiver.RECV();
    recver.CLOSE();
    DESTROY();
}

Note: this is a tightly bound pattern. For a loosely bound pattern, use ipc or tcp.

PAIR only support static parts like threads, but not dynamic nodes, because it do not (re)connect automatically.

Node Coordination

$\uparrow$ (Solution to that) eg. Pub-Sub pattern.

Add REQs at SUBs and a REP at the PUB.

• The PUB starts.
• SUBs start. When a SUB is connected to the PUB, send a REQ to the PUB. PUB counts the number of ready SUBs, and REP to them.
• When all SUBs are ready, PUB starts to distribute messages.

A more robust model could be:

• PUB opens PUB socket and starts sending “Hello” messages (not data).
• SUBs connect SUB socket and when they receive a Hello message they tell the PUB via a REQ/REP socket pair.
• When the PUB has had all the necessary confirmations, it starts to send real data.

Zero-Copy

Send and receive messages directly from and to application buffers without copying data.

void my_free (void *data, void *hint) {
    free (data);
}
//  Send message from buffer, which we allocate and ZeroMQ will free for us
zmq_msg_t message;
zmq_msg_init_data (&message, buffer, 1000, my_free, NULL);
zmq_msg_send (&message, socket, 0);

Document details:

typedef void (zmq_free_fn) (void '*data', void '*hint');

int zmq_msg_init_data (zmq_msg_t '*msg', void '*data', size_t 'size', zmq_free_fn '*ffn', void '*hint');

/*
DESCRIPTION
The zmq_msg_init_data() function shall initialise the message object referenced by 'msg' to represent the content referenced by the buffer located at address 'data', 'size' bytes long. No copy of 'data' shall be performed and 0MQ shall take ownership of the supplied buffer.

If provided, the deallocation function 'ffn' shall be called once the data buffer is no longer required by 0MQ, with the 'data' and 'hint' arguments supplied to zmq_msg_init_data().
*/

Pub-Sub Message Envelopes

Devide the subscription key and true contents, which is called “envelopes”.

Implemented by multiparts messages, send the key first with the more bit on, and then send the data.

$\to$ Key + Address of publisher + Data

High-Water Marks

Question: A sends data to B at high frequency. B encounters some other heavy tasks and can not receive data in time. This may cause A to run out of memory and crash.

Flow control solutions: set buffer size limits, and wait or drop when that happens. (Not always works)

ZMQ solution (HWM): Set a pipe for each connection, each pipe has its own HWM for sending and/or receiving. When reaching HWM, the socket will block or drop data.

Some sockets (PUB, PUSH) only have send buffers. Some (SUB, PULL, REQ, REP) only have receive buffers. Some (DEALER, ROUTER, PAIR) have both send and receive buffers.

PUB and ROUTER sockets will drop data if they reach their HWM, while other socket types will block.

Over the inproc transport, the sender and receiver share the same buffers, so the real HWM is the sum of the HWM set by both sides.

Missing Message Problem Solver

Selected notes:

• Even if you synchronize a SUB and PUB socket, you may still lose messages. It’s due to the fact that internal queues aren’t created until a connection is actually created. If you can switch the bind/connect direction so the SUB socket binds, and the PUB socket connects, you may find it works more as you’d expect.

• If you’re using ROUTER sockets, it’s remarkably easy to lose messages by accident, by sending malformed identity frames (or forgetting to send an identity frame). In general setting the ZMQ_ROUTER_MANDATORY option on ROUTER sockets is a good idea, but do also check the return code on every send call.