Hot questions for Using ZeroMQ in windows

Question:

I't trying to get a python application running on Windows and I get an ZMQError: Protocol not supported which is because ipc is not supported on windows. From what I have read the change from ipc to tcp protocol should be as easy as changing the string that is used in bind().

        master_addr = 'ipc://{0}/sailfish-master-{1}_{2}'.format(
                tempfile.gettempdir(), os.getpid(), subdomain.id)
        ipc_files.append(master_addr.replace('ipc://', ''))
        sock = ctx.socket(zmq.PAIR)
        sock.bind(master_addr)
        sockets.append(sock) 

If i change ipc:// to tcp:// I get ZMQError: Invalid argument so I guess it is not that simple. Could you walk me through the process of getting this fixed for windows, or tell me if I'm asking a stupid question.

You can see the full script https://github.com/sailfish-team/sailfish/blob/master/sailfish/master.py the code above is from line 250. SailfishCFD is a Python Lattice Boltzmann (LBM) simulation package for GPUs (CUDA, OpenCL)

Thank you very much!


Answer:

ZeroMQ is transport agnostic

That means, one may pass messages irrespective of what transport-class { inproc:// | ipc:// | tcp:// | pgm:// | epgm:// } is being used "under-the-hood".

That does not mean the same ( transport-specific ) addressing syntax will work in either case.

master_addr = 'ipc://{0}/sailfish-master-{1}_{2}'.format( tempfile.gettempdir(),
                                                          os.getpid(),
                                                          subdomain.id
                                                          )
sock.bind( master_addr )                                  # works on Linux/ipc
#   .bind( <<<tcp_addr>>> )                               # fails on "{0}{1}{2}".format-addressing"

Windows do not allow for using ipc: transport-class. Such a need-to-change will influence a bit wider scope of your source code, as there are some additional ipc-related assumptions on addressing.

As seen in:

addr         = "tcp://{0}".format( self._subdomain_addr_map[nbid] ) # tcp://<ip>:<port>?
addr         = "tcp://{0}".format( self._iface )                    # ref. #104 missing ":<port>" part!
summary_addr = 'tcp://127.0.0.1:{0}'.format( config._zmq_port )     # port free?
Start with:

framing the issue. Your code uses variable "fileName"-alike naming ( addressing ) for the IPC-pipes. There you start.

try:
     print                                   "DEBUG: Try to .bind() a ", master_addr
     sock.bind( master_addr )
     print                                   "     ==OK."

except ZMQError as Exc:
     print                                   "     ! FAILED:"
     # log & handle Exc details

except:
     print                                   "     ! FAILED: a non-ZMQ-related issue"
     # log & handle Exc details
Port#-s:

Be sure you do not command zmq.bind() on Windows to touch "priviliged" / already-used / firewall-"blocked" TCP-port#-s.

Having checked these system settings & having made zmq-call(s) syntax compatible with ZeroMQ API for tcp:// transport-class,

i.e.:

"tcp://<ip_address>:<port#>" # asString

or

"tcp://<aDnsResolvableHostNAME>:<port#>" # asString

you have it.

Question:

Service are installing properly.

But as soon as I call this LogsConsumer() code, it creates problem.

It gets installed, but doesnt start. It keeps showing "Starting" status

    protected override void OnStart(string[] args)
    {
        LogsConsumer();
    }

    public static void LogsConsumer()
    {
        using (NetMQContext ctx = NetMQContext.Create())
        {
            using (var consumer = ctx.CreatePullSocket())
            {
                consumer.Bind("tcp://localhost:5005");
                while (true)
                {
                    string msg = consumer.ReceiveString();
                }
            }
        }
    }

Answer:

OnStart should not call LogsConsumer directly. It should start a new thread which runs the LogsConsumer loop.

The Windows service control manager will not change the status from "Starting" to "Started" until the OnStart method exits. The way you have it coded, OnStart never exits so behavior you are seeing is exactly what I would expect.

BTW - once you have this problem resolved, you may run into a similar problem when you try to stop the service. You will need to modify the loop to support some way to request that it exits, then override OnStop to signal the thread to exit. The pure ZeroMQ way to handle this is to have the thread poll two sockets - your existing external bound socket plus a second inproc socket (see http://zguide.zeromq.org/page:all#Handling-Multiple-Sockets).

The purpose of the second socket is to allow the OnStop method (which is running in a separate service thread) to send a message to your worker thread to stop it.

Question:

I want to use ZeroMQ for Windows 10 64bit to use in Java.

I am using Intelj as my IDE. I googled and the only thing I found out was to compile library from source ( Mostly for Linux ).

Is there any way to use a compiled ZeroMQ library, so I can import it easily in my java code?


Answer:

To use zeromq in java you have 2 choices, 1. Use jeromq which is a java implementation of zeromq, and 2. Use jzmq which is a jni binding required libzmq installed on your system. If you are willing to use jzmq you can download and install libzmq from here

Question:

So, I have a process that manages a bunch of workers that uses a REQ/REP and a PUB/SUB pattern for each worker bee. I set the monitor interval to 250 [ms] and all has been working fine.

When I deploy onto a Windows server, and startup the resource monitor, the amount of network traffic (bytes written) by this node process is upwards of 64-100 [MB/s] and that doesn't include any real application transactions yet, because the traffic is there whether I start the children or not, and it doesn't abate when the children do come online.

Suspecting the interval setting I played with increasing the value, and do notice that higher interval values (less often) does lead to a decrease in the traffic, but the cost of that is it takes that long to detect connect/disconnect events.

Now I haven't tried to narrow it down yet to whether it is actually the REQ/REP pattern or the PUB/SUB pattern that is the issue, but I am curious what part of the documentation I overlooked that would explain this behavior.

We are probably going to swap out the ZeroMQ in favor or GRPC next release, but I have questions to answer now.

I believe I have tagged everything relevant to the operating environment.


Answer:

When I deploy onto a Windows server, and startup the resource monitor, the amount of network traffic (bytes written) by this node process is upwards of 64-100 [MB/s]

I guess you are already pretty aware of how the low-level ZeroMQ native API tools actually work down there, under the hood. If not, may read how the native API implementation under the calls to zmq_socekt_monitor() itself sets another layer of ZeroMQ pipes-and-plumbing, so as to become able to monitor its own activities ( events et al ) and review, how well do Windows ( native or virtualised platforms ) cope with using inproc://-channels setup amongst your Context()-instance(s)' pool-of-IOthread(s) and the said bunch of bees. Try to tweak the number of IO-threads and may be better map the ZMQ_AFFINITY in a different manner, so as to split the " background " workloads

...Each call to this method creates a ZMQ_PAIR socket and binds that to the specified inproc:// endpoint. To collect the socket events, you must create your own ZMQ_PAIR socket, and connect that to the endpoint. The events argument is a bitmask of the socket events you wish to monitor, see Supported events below. To monitor all events, use the event value ZMQ_EVENT_ALL. Each event is sent as two frames. The first frame contains an event number (16 bits), and an event value (32 bits) that provides additional data according to the event number. The second frame contains a string that specifies the affected TCP or IPC endpoint.

This is my suspect #1 for the root-cause of seeing ~ 100[MB/s] of traffic @ 250 [ms] cadence, as reported above ( having no MCVE, as you have explained above ).

Turning on all possible events ( in the herd ) may generate indeed some volume of traffic, as each FSA-event reports a colourfull state since launched, irrespective of the intended PUB/SUB + REQ/REP Scalable Formal Communication Archetype patterns' state, covering all the intended ZeroMQ infrastructure's connectivity lifecycle, propagating each such configured monitored-event during either of { setup | operations | termination | release}-phase:

{ ZMQ_EVENT_CONNECTED,
  ZMQ_EVENT_CONNECT_DELAYED,
  ZMQ_EVENT_CONNECT_RETRIED,
  ZMQ_EVENT_LISTENING,
  ZMQ_EVENT_BIND_FAILED,
  ZMQ_EVENT_ACCEPTED,
  ZMQ_EVENT_ACCEPT_FAILED,
  ZMQ_EVENT_CLOSED,
  ZMQ_EVENT_CLOSE_FAILED,
  ZMQ_EVENT_DISCONNECTED,
  ZMQ_EVENT_MONITOR_STOPPED
  }

so data flow even before the first intended ZeroMQ infrastructure's .connect() has appeared.


Epilogue:

In case there are also the said [GB/s] flows reported on the L3+ network layers, the SIGINT experts may better request due dilligence and clarifications from the platform vendor(s), what has been behind such a platform's self-reporting ( if not even a back-door class ) practices, that may explain, sure - better stop, such remarkable network exgress data-flows. Did someone dare to say the platform might have been hacked? Well, hope it was not.

Question:

I'm trying to write a client/server application with ZeroMQ. I've written a server which sends two messages via zmq_send() one right after the other in response to a GET_DATA message from the client.

The client sends GET_DATA and then tries to call zmq_recv() twice to get both messages sent by the server. Unfortunately the second call to zmq_recv() fails in the client with a -1 return and no errno.

In response to GET_DATA the server simply calls:

zmq_send (responder, "**start**", 9, 0); 
zmq_send (responder, "**done**", 8, 0);

Here is the code of the client:

#include "stdafx.h" 

#include "zmq.h" 

int _tmain(int argc, _TCHAR* argv[]) 
{ 
    printf ("Connecting to server.\n"); 
    void *context = zmq_ctx_new (); 
    void *requester = zmq_socket (context, ZMQ_REQ); 
    zmq_connect (requester, "tcp://localhost:5555"); 

    char szCommand[] = "GET_DATA"; 
    zmq_send(requester, szCommand, sizeof(szCommand), 0); 

    char buffer[512]; 
    int nLen = zmq_recv(requester, buffer, sizeof(buffer) - 1, 0); 
    if( nLen >= 0 ) printf("%*.*s\n", nLen, nLen, buffer); 
    nLen = zmq_recv(requester, buffer, sizeof(buffer) - 1, 0); 
    if( nLen < 0 ) printf("errno = %s\n", strerror(errno)); 
    printf("done\n"); 

    zmq_close (requester); 
    zmq_ctx_destroy (context); 
    fgetc(stdin); 
    return 0; 
}

The output of the client is:

Connecting to Server.
**start**
errno = No Error
done

Can someone please explain why the second call to zmq_recv() returns -1 in the client?

Note: in a previous version I was getting errno of 9 (which seems to be bad file or something).

EDIT for posterity: This was written on Win32 which should use zmq_errno() instead of accessing errno directly but I only learned that later, so this problem did cause some noise here.


Answer:

Req-rep socket types in zeromq is following a state-machine, so you can request and than must receive, in your case you receive twice which is not supported. However you can use multipart messages, on the server side, on the first send call with ZMQ_SNDMORE in the flags parameter.

Like so:

zmq_send(response, "Part1", 5, ZMQ_SNDMORE);
zmq_send(response, "Part2", 5, 0);

Question:

I've been developing an app for Windows Phone 8.1 and i wanted to use ZeroMQ for the communication with the server. I've used ZMQ before and it has always worked after some trying but i just coulnd't get it to work with WP8.1. I already tried to use NetMQ but i get the error "the package does not contain any assembly references or content files that are compatible with that framework" when trying to install it via NuGet, the same Problem with clrzmq4 and Castle.ZMQ Is there any possibility out there to get ZeroMQ running on WP8.1? I'm thankful for your help.


Answer:

According to this thread, ZeroMQ is not compatible with Windows Phone, since this doesn't support the .NET sockets library.

I guess you'll have to use an alternative or build your own then...

Question:

I'm trying to get ZeroMQ up and running on Windows 10, my aim to use ZeroMQ/flatbuffers to exchange structured messages between a Java process and a Python process. I'm find the ZeroMQ website terse and it's not clear to me how to simply get ZeroMQ libs built with ease, the steps described always result in an error of some sort.

Can anyone point me towards a simple set of instructions to get the libraries generated that actually works ?


Answer:

There are faster options

Building the libs:

  1. download and install Community Edition with package "Individual components">"C++ CMake tools for Windows"
  2. download latest ZeroMQ release as .zip
  3. unzip
  4. open Visual Studio
  5. choose "Open a local folder" and select ZeroMQ folder
  6. right click CMakeLists.txt > CMake settings for ZeroMQ > x86-Release
  7. above the project tree select "CMake targets view" from "Toggle between Solution and Folder views"
  8. right click ZeroMQ Project > Build All
  9. your user profile folder will contain a folder CMakeBuilds which contains the build results

Question:

I am trying to install ZeroMQ in my windows.

I have xampp and php version 7.0, so what are the best steps to install this library?


Answer:

Try this

You can directly download and install .exe file