Hot questions for Using ZeroMQ in ios

Question:

I am developing an app for iOS and Android. The basic functionality is to keep a certain set of data synchronized across all devices in a Wi-Fi network without a central server. Every device can modify that set of data.

The current approach is to discover the other devices via Bonjour/Zeroconf and then send the "change messages" to all devices via ZeroMQ.

As both Frameworks cause quite a lot of problems to implement I am asking if this is the correct way to accomplish this.

I had most of the logic implemented with Bonjour and HTTP-Requests sent to all devices. The problem was simply network requests which would not get received even after three tries because the network failed. I want to have some kind of reconstruction of a general state or a more reliable messaging framework.

Would some kind of Gossip approach to spread the information as well as discover all devices be better?


Answer:

Simple schemes do not match all requirements right from the box.

Neither an ad-hoc role-asymmetry of becoming a Server-on-demand so as to decide upon a change as in "Survey (Everybody Votes)" ( Fig.-s courtesy nanomsg.org )

or a Coalition-of-Nodes role-symmetry in a "Bus Pattern" meets all the requirements per-se.


A Continuous Discovery "Phase"

is a task to be operated as a continuous self-identification, so as to yield the Coalition-of-Nodes the relevant set of information for whom to wait during the voting and for whom not. Reciprocally, when it is fair for the to broadcast <aListITEM> change and expect a voting to be supported by it´s Coalition-of-Nodes "neighbourhood".

Pieter Hintjens' 400+ pages book The ZeroMQ Guide - for Python Developers, Chapter 8.3 will give you some initial insights on autonomous preemptive and/or cooperative discovery plus some remarks on WiFi are in previous chapters. Also kindly note a closing remarks on ISO-OSI-L2/L3 uncertainties in >>> Limitations on WiFi SSID L3 ARP based discovery


Methods for <aListITEM> change propagation accross the current Coalition-of-Nodes

is just another sub-protocol ( layer ) to be implemented inside the Coalition-of-Nodes.

Does a Bus or some hybrid Scaleable Formal Communication Pattern with "Survey" Voting meet all requirements?

Maybe yes, maybe no.

First list all requirements to be able to Design "against" such a mandatory feature set.

Second, validate, that the feature set is legitimate and feasible for each and every Node, that will dynamically become / cease to be a member of the Coalition-of-Nodes.

Third, design the non-blocking, self-recovering community -- a higher-order-FSA-of-FSAs -- with adequate handshaking, re-synchronisation / watchdog / timeout(s) and propagation of <aListITEM> updates & voting mechanisms, so that these meet the mandatory design features set.

Do not rely on ready-made primitives ( at a cost of "bending" mandatory design feature set so as to meet the library-available primitive, but rather develop another, a new, higher-order Formal Communication Pattern Signalling, being assembled from library primitives, so that it meets the whole specification. )

Question:

We're working with a latency-critical application at 30fps, with multiple stages in the pipeline (e.g. compression, network send, image processing, 3D calculations, texture sharing, etc).

Ordinarily we could achieve these multiple stages like so:

[Process 1][Process 2][Process 3]

---------------time------------->

However, if we can stack these processes, then it is possible that as [Process 1] is working on the data, it is continuously passing its result to [Process 2]. This is similar to how iostream works in c++, i.e. "streaming". With threading, this can result in reduced latency:

[Process 1]
    [Process 2]
        [Process 3]
<------time------->

Let's presume that [Process 2] is our a UDP communication (i.e. [Process 1] is on Computer A and [Process 3] is on Computer B).

The output of [Process 1] is approximately 3 MB (i.e. typically > 300 jumbo packets at 9 KB each), therefore we can presume that when we call ZeroMQ's:

socket->send(message); // message size is 3 MB

Then somewhere in the library or OS, the data is being split into packets which are sent in sequence. This function presumes that the message is already fully formed.

Is there a way (e.g. API) for parts of the message to be 'under construction' or 'constructed on demand' when sending large data over UDP? And would this also be possible on the receiving side (i.e. be allowed to act on the beginning of the message, as the remainder of the message is still incoming). Or.. is the only way to manually split the data into smaller chunks ourselves?

Note:the network connection is a straight wire GigE connection between Computers A and B.


Answer:

TLDR; - simple answer is no, ZeroMQ SLOC will not help your project win. The project is doable, but another design view is needed.

Having stated a minimum set of facts:- 30fps,- 3MB/frame,- 3-stage processing pipeline,- private host-host GigE-interconnect,

there is not much to decide without further details.

Sure, there is a threshold of about 33.333 [ms] for the pipeline end-to-end processing ( while you plan to lose some 30 [ms] straight by networkIO ) and the rest you leave to designer's hands. Ouch!

Latency controlled design shall not skip a real-time I/O design phase

ZeroMQ is a powerhorse, but that does not mean, it could save a poor design.

If you spend a few moments with timing constraints, the LAN networkIO latency is the worst enemy in your view.

Ref.: Latency numbers everyone should know

1 ) DESIGN Processing-phases2 ) BENCHMARK each Phase's implementation model3 ) PARALLELISE wherever Amdahl's Law says it is meaningful and where possible

If your code allows for a parallelised processing, your plans will get much better use of "progressive"-pipeline processing with a use of a ZeroMQ Zero-copy / ( almost ) Zero-latency / Zero-blocking achievable in inproc: transport class and your code may support "progressive"-pipelining as you go among multiple processing phases.

Remember, this is not a one-liner and do not expect a SLOC to control your "progressive"-pipelining fabrication.

[ns] matter, read your numbers from data-processing micro-benchmarks carefully.They do decide about your success. Here you may read how much time was "lost" / "vasted" in just changing a color-representations, which your code will need in object detection and 3D scene-processing and texture post-processing. So have your design criteria set for rather high standard levels.

Check the lower left window numbers about milliseconds lost in this real-time pipeline.

If your code's processing requirements do not safely fit into your 33,000,000 [ns] time-budget with { quad | hexa | octa }-core CPU resources and if the numerical processing may benefit from many-core GPU resources, there may be the case, that Amdahl's Law may well justify for some asynchronous multi-GPU-kernel processing approach, with their additional +21,000 ~ 23,000 [ns] lost in initial/terminal data transfers +350 ~ 700 [ns] introduced by GPU.gloMEM -> GPU.SM.REG latency masking ( which happily has enough quasi-parallel thread-depth in your case of image processing, even for a low-computational density of the expected trivial GPU-kernels ) Ref.: GPU/CPU latencies one shall validate initial design against:

Question:

I want to use ZeroMQ from my cross-platform application that will target Windows, Android and iOS.

But none of the C# implementation that I found is PCL, so I cannot use them for this purpose.

Should I implement a P/Invoke solution to use it.

Is there any solution for this problem?


Answer:

When there is no PCL version, I'd throw the code of the .NET implementation into a shared library and include them in iOS, Android, and Windows. The libraries there aren't as limited as a PCL that targets these 3 platforms. If it doesn't compile, you could have a look at the problems and try to eliminate them by replacing them. That's why you should choose a full .NET implementation without a lot of dependencies (like https://github.com/zeromq/netmq).

Question:

I want to develop an iOS app which should be able to send messages to other WiFi direct enabled devices (no matter if it is an iOS device or Android device) and receive messages from them.

I found a suitable library ZeroMQ which can be used to accomplish the task. I have compiled the library for iOS successfully.

I reffered szehnder's sample code but the app doesn't do anything. It shows a blank table view only. I checked in the code and found that the code to receive a message is present there but the method to send the message is empty.

How can we detect other WiFi direct devices? Send and receive messages to them? Any pointers are highly appreciated.

As suggested by Jason, I am adding whatever I have achieved till now: I have an android app which connects to other android devices using WiFi direct and sends and receive messages from them, and uses ZeroMQ. I need to communicate with that app with my iOS app.

Below is my code to receive the messages:

-(void) setupSockets:(ZMQContext*)myContext {

 ZMQSocket *subscriber = [myContext socketWithType:ZMQ_SUB];
[subscriber setData:nil forOption:ZMQ_SUBSCRIBE];
[subscriber connectToEndpoint: [NSString stringWithFormat:@"tcp://%@:%@", @"192.168.49.1", @"26665"]];

while (1) {

    NSData *m = [subscriber receiveDataWithFlags:0];
    if (m!=NULL) {
         NSString *s = [NSString stringWithUTF8String:[m bytes]];
        NSLog(@"Message: %@", s);

        [self performSelectorOnMainThread:@selector(postMessageNotification:) withObject:s waitUntilDone:NO];

    }
    sleep(3);

}

}

Here 192.168.49.1 is the ip address of the host android device and 26665 is the port number used to receive messages. The control reaches up to call of receiveDataWithFlags: successfully but does nothing after that.

Now to send the data:

-(IBAction) sendMessageClicked:(id)sender {
NSString *pubEndPoint = [NSString stringWithFormat:@"tcp://%@:%@", @"192.168.49.1", @"26666"];
ZMQSocket *pubSocket = [objMeeps.context socketWithType:ZMQ_PUB];
[pubSocket connectToEndpoint:pubEndPoint];

NSData *msg = [@"hi" dataUsingEncoding:NSUTF8StringEncoding];
   [pubSocket sendData:msg withFlags:ZMQ_NOBLOCK];
}

The method gives an error message ZMQLogError: *** <ZMQSocket: 0x17405fdd0 (ctx=0x174029480, type=ZMQ_PUB, endpoint=tcp://192.168.49.1:26666, closed=0)>: zmq_send: Resource temporarily unavailable. When I searched regarding this error I got that the error is occurring due to absence of any receiver but my iPhone is connected to the WiFi network created by the android host using WiFi Direct.

I am new to ZeroMQ so might be missing something very small. Hope this clarifies the issue enough.Please guide me regarding this.


Answer:

So you say tcp://192.168.49.1:26666 is the address of your Android host, but so far as I can tell you're trying to publish and subscribe from your ios device, at least for this example. So, let's be explicit: for this example, ignore your Android device and just try to get your ios application talking to itself.

Step 1: use the ip:port for your ios device

ZMQ has two sides to every connection, one side that "binds" (and could be considered the server) and one side that "connects" (and could be considered the client). Your choices are flexible, but for the sake of example it's probably best for your publisher to bind and your subscriber to connect. Right now, you have both sides connecting. This is likely the cause of the error you're seeing.

Step 2: change your pub socket to bind on its address rather than connect

When using PUB/SUB, you must subscribe to the "topics" you want. In your example code you're subscribing to "NASDAQ". For you to receive messages, the text of the message must begin with your subscriber topic, for instance "NASDAQ:hi" would get through your subscription filter.

Step 3: change your zmq_subscribe option to subscribe to an empty string "" to get every message that comes through

Making those three changes should put you on the right track. I highly suggest you read the guide, it covers these sorts of basic usage guidelines and has a fair body of ios example code to boot.

Question:

I want to implement FileMQ for file transfer from iOS to android in my iOS app. I tried the steps given here but it causes errors at many steps. Also I need to know how should I use the downloaded library.

How should I compile FileMQ for iOS and use it?

Any information in this regard is appreciated!


Answer:

I downloaded a C version of the library from the link mentioned in the question and compiled it on linux machine. I made some changes in the headers to make it iOS compatible as the headers were generated for linux. Now I am using the same copy in my iOS project.