Hot questions for Using ZeroMQ in architecture

Top 10 C/C++ Open Source / ZeroMQ / architecture

Question:

I want to create a Publish / Subscribe architecture, using CZMQ-4.0.2, but I am not able to understand how to use the new zsock APIs.

Can anyone point me to some examples using the new APIs?


Answer:

tldr;

Examples are on the bottom of the site

Little explanation

I'm assuming that you're asking for CZMQ specific usage, not how to use ZeroMQ sockets, and what are the quirks of PUB/SUB pattern.

When using CZMQ you don't need to worry about context, it is done internally. zsock_new functions family returns pointer to zsock_t, an opaque identifier for socket. You need to remember to call zsock_destroy(&socket) when you're done with it, to avoid memory leaks.

In most common usage you don't need to worry about connecting and binding because zsock_new_XXX takes care of that. To know what action was taken you can look to manual.

//  Create a PUB socket. Default action is bind.
CZMQ_EXPORT zsock_t *
    zsock_new_pub (const char *endpoint);
//  Create a SUB socket, and optionally subscribe to some prefix string. Default
//  action is connect.
CZMQ_EXPORT zsock_t *
    zsock_new_sub (const char *endpoint, const char *subscribe);

If you're planning to do some unusual binding/connecting, you can add a prefix to endpoint. @ indicates bind, > connect.

zsock_t *sock = zsock_new_push("@ipc://test");

Now, to send message you can use plenty of methods (zsock_send, zmsg_send, zstr_send, zstr_sendx, zstr_sendf, zframe_send), most generic is zsock_send. It has printf like prototype, where you need to pass a picture of a message. Each char in this string represents single frame in message (or more frames because you can also pass another message). It is described in here:

//  Send a 'picture' message to the socket (or actor). The picture is a
//  string that defines the type of each frame. This makes it easy to send
//  a complex multiframe message in one call. The picture can contain any
//  of these characters, each corresponding to one or two arguments:
//
//      i = int (signed)
//      1 = uint8_t
//      2 = uint16_t
//      4 = uint32_t
//      8 = uint64_t
//      s = char *
//      b = byte *, size_t (2 arguments)
//      c = zchunk_t *
//      f = zframe_t *
//      h = zhashx_t *
//      U = zuuid_t *
//      p = void * (sends the pointer value, only meaningful over inproc)
//      m = zmsg_t * (sends all frames in the zmsg)
//      z = sends zero-sized frame (0 arguments)
//      u = uint (deprecated)
//
//  Note that s, b, c, and f are encoded the same way and the choice is
//  offered as a convenience to the sender, which may or may not already
//  have data in a zchunk or zframe. Does not change or take ownership of
//  any arguments. Returns 0 if successful, -1 if sending failed for any
//  reason.
CZMQ_EXPORT int
zsock_send (void *self, const char *picture, ...);

One what may be unclear is this void *self, it is actually our zsock_t * returned from zsock_new. In prototype, it is declared as void * because this function also accepts zactor_t *.

Important: Does not change or take ownership of any arguments.. You need to free/destroy data after sending.

Receiving looks very similar. It is like sscanf, and zsock_recv creates objects, so again, you need to take care of memory.

Big difference in behavior between ZeroMQ and CZMQ is LINGER socket option. For ZeroMQ it was infinite (-1), where CZMQ has a default value of 0 (no blocking). So anytime when you'll have zsock_send followed by zsock_destroy, your message may not be delivered. Linger value can be set individually for the socket by using zsock_set_linger, or globally zsys_set_linger.

Example of Publisher

#include <czmq.h>

int main(int argc, char ** argv) {
  zsock_t *socket = zsock_new_pub("ipc://example.sock");
  assert(socket);

  while(!zsys_interrupted) {
    zsys_info("Publishing");
    zsock_send(socket, "sss", "TOPIC", "MESSAGE PART", "ANOTHER");
    zclock_sleep(1000);
  }

  zsock_destroy(&socket);
  return 0;
}

Example of Subscriber

#include <czmq.h>

int main(int argc, char ** argv) {
  zsock_t *socket = zsock_new_sub("ipc://example.sock", "TOPIC");
  assert(socket);

  char *topic;
  char *frame;
  zmsg_t *msg;
  int rc = zsock_recv(socket, "sm", &topic, &msg);
  assert(rc == 0);

  zsys_info("Recv on %s", topic);
  while(frame = zmsg_popstr(msg)) {
    zsys_info("> %s", frame);
    free(frame);
  }
  free(topic);
  zmsg_destroy(&msg);

  zsock_destroy(&socket);
  return 0;
}

Question:

I am trying to determine the exact behaviour and potential limitations of the so-called 'Extended Pub-Sub architecture' from the ØMQ guide.

XPUB and XSUB are described:

We need XPUB and XSUB sockets because ZeroMQ does subscription forwarding from subscribers to publishers. XSUB and XPUB are exactly like SUB and PUB except they expose subscriptions as special messages. The proxy has to forward these subscription messages from subscriber side to publisher side, by reading them from the XSUB socket and writing them to the XPUB socket. This is the main use case for XSUB and XPUB.

I've set up a the XSUB and XPUB sockets as the front-end and back-end of a proxy, and have another pair of PAIR sockets wired into the capture port. This allows me to observe the messages being passed through the proxy.

In my architecture each node is both a PUB and a SUB. Essentially I'm hoping this XPUB/XSUB proxy will provide a shared bus, with topic-prefix subscriptions.

After a SUB node connects, it must subscribe with a (potentially empty) topic. This causes a one-frame message to be transmitted through the proxy. Assuming my topic is {0xff 0xFF}, the message is:

{0x01 0xFF 0xFF}

The leading 0x01 indicates a subscription, followed by the topic bytes. A similar message with a 0x00 instead of a 0x01 represents an unsubscribe.

What I am concerned about is where exactly the subscription information is kept in this architecture.

According to the guide:

From ZeroMQ v3.x, filtering happens at the publisher side when using a connected protocol (tcp:// or ipc://).

If filtering is indeed performed on the publisher side, then is it a problem if a SUB subscribes before a PUB comes online? Will the PUB be informed of pre-existing subscriptions, perhaps from XSUB?

My system will have nodes with dynamic lifetimes. Will this be an issue, and are there any other issues I should be aware of?


Answer:

If filtering is indeed performed on the publisher side, then is it a problem if a SUB subscribes before a PUB comes online?

No, that will work itself out.

Will the PUB be informed of pre-existing subscriptions, perhaps from XSUB?

Yes exactly.

My system will have nodes with dynamic lifetimes. Will this be an issue, and are there any other issues I should be aware of?

You'll lose published messages that have no subscribers, so either create a proxy that windows published messages or make subscribers ask publishers to re-publish and have idempotent handling of messages.

Here's a sample of a full-duplex proxy that you can play around with. You'll note that if you start the "pong" (the wall that the ball bounces on) before "ping" (the publisher of the ball), it will all work, but if you publish a ping before the pong subscriber is started, it will be lost.

Question:

I am trying to create a network architecture which has a single server and multiple clients. The clients can connect and disconnect at any time so they need to announce their existence or shut-down to the server. The server must be able to send data to any particular client.

What is the best scalability protocols/architecture to use for this?

Currently I use a REQ/REP so that clients can 'login' and 'logout', and a SURVEY socket so that the server can send data to all clients. The message sent has an ID for the particular client it wants to process the message.

Is this good, or is there something better?


Answer:

Sounds more like you need publisher subscriber. With both 0MQ and nanomsg you don't need to do anything in particular to manage connection / disconnection, the library does that for you.

However if you want more sophisticated message management (such as caching outgoing messages just in case another client chooses to connect) then you will have to manage that yourself. You might use a single push pull from the clients for them to announce their presence (they'd send a message saying who they were), followed by more push pulls from the server to each of the clients to send the messages from the cache that you have. Fiddly, but still a whole lot easier than programming up with raw sockets.

Using req rep can be problematic - if either end crashes or unexpectedly disconnects the other can be left in a stalled, unrecoverable state.

Question:

I need to make an architecture decision on a cross-platform app-suite. I basically want to try new way of decoupling modules and implement network I/O using ZeroMQ, knowing it's a message queue for in-process, inter-process and networking applications. But I'm not sure how it can fit in with my case.

I'd appreciate it if anyone could clarify a few things before I spend the next week reading their intriguing book: http://zguide.zeromq.org/page:all

I've checked these questions but didn't get my answers:

  • How to use zeroMQ in Desktop application
  • How to use ZeroMQ in an GTK/QT/Clutter application?

My requirements:

  • Desktop hosts on Windows and macOS, as separated console backend and GUI frontend; the backend must be written in C++;
  • Mobile guests on iOS and Android, backend written in C++;
  • Desktop talks with mobile using TCP;

Old Way

As for the desktop backend (the console app), a few years back, my first step would be writing a multithreaded architecture based on Observer/Command patterns:

  • Set the main thread for UI and spawn a few threads.
  • One "scheduler" thread for message handling: a queue to get notifications from other modules and another queue for commands. Each command type introduces its own dependencies. The scheduler pumps messages and issues commands accordingly.
  • Other "executor" threads for device monitoring, multiplex network I/O between one desktop and multiple mobile devices, all sending messages to scheduler to have real work scheduled.

I would then need to implement thread-safe message queues, and will inevitably have coupling between schedulers and a bunch of Command classes that are essentially just function wrappers of those executors' behaviors. With C++, this would be a lot of boilerplate code.

New Way to Validate

But it's 2019 so I expect less hand-written code and would try something new. With ZeroMQ, I'd love to see if my expectation holds. I'd love to ...

  • Remove the need of writing a scheduler and message/command queues from scrach, by just passing ZeroMQ requests between in-process modules across threads, because writing scheduling from scratch is tedious and unproductive.
  • Simplify network I/O between desktop and mobile devices. For this part I've tried ASIO and it wasn't significantly more convenient than raw socket and select, plus it's C++-only.
  • Decouple GUI and console app with ZeroMQ-based IPC, so that GUI can be rewritten using different technologies in various languages.
  • Perceive low-latency for both desktop and mobile users.

Is my expectation reasonable?


Answer:

If new to ZeroMQ domains, feel free to review this and best enjoy a very first look at "ZeroMQ Principles in less than Five Seconds" before diving into further details


An above referred post has presented an assumption, that:

ZeroMQ is based on the assumption that there is an while (1) ... loop that it is inserted into

is completely wrong and misleading any Architecture planning / assessment efforts.

ZeroMQ is a feature-rich signaling/messaging metaplane, that is intended to provide a lot of services for the application-level code, that may enjoy a light-weight re-use of the smart, complex on low-level, efficient handling of signaling/messaging infrastructure, be it used for in-process, inter-process and inter-node multi-agent distributed fashion, using for that goal many already available transport-class protocols:

{ inproc:// | ipc:// | tipc:// | vmci:// | tcp:// | pgm:// | epgm:// | udp:// }


This said, let's follow your shopping-list :

My requirements:

  • c++ ZeroMQ: [PASSED] Desktop hosts on Windows and macOS, as separated console backend and GUI frontend; the backend must be written in C++;
  • c++ ZeroMQ: [PASSED] Mobile guests on iOS and Android, backend written in C++;
  • tcp ZeroMQ: [PASSED] Desktop talks with mobile using TCP;

I'd love to ...

  • Remove the need of writing a scheduler and message/command queues from scrach, by just passing ZeroMQ requests between in-process modules across threads, because writing scheduling from scratch is tedious and unproductive.
  • Simplify network I/O between desktop and mobile devices. For this part I've tried ASIO and it wasn't significantly more convenient than raw socket and select, plus it's C++-only.
  • Decouple GUI and console app with ZeroMQ-based IPC, so that GUI can be rewritten using different technologies in various languages.
  • Perceive low-latency for both desktop and mobile users.

Is my expectation reasonable?

Well :

  • there is obviously no need to write scheduler+Queues from scratch. Queue-management is built-in ZeroMQ and actually hidden inside the service-metaplane. Scheduling things among many-actors is on the other hand your design-decision and has nothing to do with ZeroMQ or other technology of choice. Given your system-design intentions, you decide the way ( "autogenerated magics" are still more a wishful thinking than any near-future system design reality )

[PASSED] QUEUES : built-in ZeroMQ [NICE2HAVE] SCHEDULER : auto-generated for any generic distributed many-agent-wide ecosystem (yet, hard to expect in any near future)

  • network ( and any in principle ) I/O is simplified already in the ZeroMQ hierarchy of services

[PASSED] : SIMPLIFIED NETWORK I/O - ZeroMQ provides already all abstracted Transport-Class related services hidden to the transparent use of the signaling/messaging metaplane,so the application code enjoys to "just" { .send() | .poll() | .recv() }

[PASSED] : Decoupling GUI from any other part of the ParcPlace-Systems-pioneered-MVC-architecture. Using this since ZeroMQ v2.11 for a (far)remote keyboard over TCP/IP network and even possible to integrate into actor-based GUI, like Tkinter-GUI actors may well serve this distributed local-Visual/remote-distributed-Controller/remote-distributed-Model. If mobile-terminal O/S introduces more complex constraints on the local-Visual MVC-component, proper adaptations ought be validated with domain-experts on that particular O/S properties. ZeroMQ signaling/messaging metaplane has not been considered so far to contain any constraints per se.

[PASSED] : LATENCY - ZeroMQ was designed from the very start for delivering ultimately low-latency as a must. Given it can feed HFT-tranding ecosystems, the Desktop/Mobile systems are orders of magnitude less restrictive in the sense of E2E lump sum accumulation of all the visited transport + O/S-handling latencies.

Question:

So, there we've got a client-server interaction via ZMQ and have stucked into an architectural arguing about the proper pattern fitting our needs. I hope that someone wise and experienced may help resolve it.

It's not essential, but I need to mention that ZMQ is not being used directly, it's rather a Qt binding with C++ over the library itself, so low-level customizations are possible but undesirable (they would increase implementation efforts significantly).

Current architecture

There was a need of some reliable, convenient & robust API broker, the one's been implemented via REQ <-> REP: statuses, error codes, encryption, etc. Encryption 's been implemented via separate authorization SSL channel, providing API with session keys, it's mentioned here to empasize that as far as SSL has not been provided at ZMQ's socket level (looked too complex), "session keys" exist (symmetric encryption key per client), it limits pattern solutions somehow.

So, there exist requests (client) + responses (server), it works. But we've recently met a need in some notification mechanizm for clients. Let's say the current broker API provides some types of data: X, Y, Z (lists of something). The client can get any of them but it has to be notified when any changes in X or Y or Z occur in order to know that new requests are to be done.

The problem

Obviously, clients should receive either data updates or notifications that such updates exist. It could be a mere PUB-SUB problem, but it seems almost impossible to make this solution encrypted, or at least authorization-aware (not mentioning really "crutchy" ways to do it).

After some discussion two opinions appeared, describing two different workarounds:

  1. Still use PUB-SUB, but only send notification type to the subscribers, like "hey, there's new X present". Clients-subscribers would have to perform already implemented API requests (REP-REQ) with session keys and all. Advantages: easy and working. Disadvantages: client logic complication.
  2. Just rewrite API to use couples of ZMQ_PAIR sockets. Results in client-server behavior similar to plain sockets, but notifications can be "sent back" from server. Advantages: simple scheme. Disadvantages: rewriting, also broker won't differ much from a simple socket solution.
Question

What would you adwise? Any of the descibed solutions or something better, maybe? Possibly X-Y problem exists here? Maybe something is considered a common way of solving problems like that?

Thanks in advance for any bright ideas.


Answer:

ZMQ_PAIR socket are mainly used for communication between threads so I do not think they are a good solution for a client/server setup, if it all possible.

You could use ROUTER/DEALER instead of REQ/REP as they allow other patterns than just request/reply. I think newer version of ZeroMQ provide SERVER/CLIENT as a better alternative but I have not used them myself.

However, I would prefer a solution with a separate PUB/SUB channel because:

  • You don't have to change the existing REQ/REP protocol.
  • It allows only clients that are interested in notifications to connect to the PUB socket and process notifications. Other clients can use just the existing REQ/REP protocol.
  • PUB/SUB can automatically send notification to multiple clients.
  • PUB/SUB allows subscribing to specific notifications only.