Hot questions for Using ZeroMQ in mql4

Question:

this is somehow intended to (maybe) be(come) a guideline post in terms of MetaTrader 4/5 and its corresponding language MQL4 both set into context with sending data to external servers.

In my special case I am building a Django/Python based web application that will process FOREX trading data for further use.

Thus, I am in search for a proper solution to send data from the MetaTrader 4/5 Terminal to an external server periodically (e.g. every 60 seconds) formatted as json or csv (if possible). In particular, the data to be sent is the accounts trade history and the running + pending trades set.

After doing the research I basically found the following approaches:

1.) Using webrequest() in MQL4 wrapped in an expert advisor

As the official MQL4 docu suggests, the webrequest() function sends an http:// request to a specified server. This is a related SO thread:How to post from MetaTrader Terminal 5 MQL 5 a request to my nodejs server, which is running locally on my MT5 host?

and the official documentation:

https://docs.mql4.com/common/webrequest

This could be wrapped into an expert advisor to execute the request periodically on defined events given.

Which kind of data from the MT4/5 terminal can be populated into the data array? How can that data be formatted? Is it possible to format it as json straight away or should that be done on the server-side?

2.) Using ZeroMQ

This is a setup I found in this thread: How to send a message in MQL4/5 from MetaTrader Terminal to python using ZeroMQ?

How would this be accomplished within the MetaTrader environment? Will this still be an expert advisor or some kind of DLL solution? What is the ZeroMQ's role within the setup?

What are the pros and cons compared to the webrequest() function?

3.) Others?

Are there any other possible approaches on achieving this like e.g. with APIs or MQL4 scripts?

Since this is a rare topic, I am loooking forward to any however small idea and input.


Answer:

Since this is a rare topic, ... is it, indeed, so rare to need this ?

Welcome to the club - my above cited answer has received within 1,5 years Zero-votes so far

Nevertheless,having used ZeroMQ since v2.11+, thanks to the R&D work published by Austen CONRAD - all thanks and deep respect for his persistence we owe to him.

Q : "How would this be accomplished within the MetaTrader environment?"

One simply #import-s the DLL and starts to use ZeroMQ-API-wrapper calls. A few details went a bit more complex, after MetaTrader has silently changed internal representation of string to cease to be a string ( becoming a struct in "New"-MQL4.56789, but you will learn how to live with this "Always On Watch"-style, so as to survive in production )

Q : "Will this still be an expert advisor or some kind of DLL solution?"

ZeroMQ can be used in either and/or all of: - Expert Advisor-type of MQL4-code - Custom Indicator-type of MQL4-code - Script-type of MQL4-code and can even provide a proxy-signaling/messaging-layer, so as to communicate among these otherwise separate and not-cooperating processes inside the MT4-Terminal Ecosystem.

Example: I have MT4-Terminal processes cooperate with external AI/ML-based market analyser, which auto-detects windows-of-opportunity plus having an external CLI-console, as a remote-keyboard for MT4-Terminal hosted Control Panel, showing system health-state and listening to a remote-keyboard, used for a remote CLI-command control ( for both configuration and maintenance tasks of the whole multi-party distributed trading system )

Q : "What is the ZeroMQ's role within the setup?"

ZeroMQ provides an independent, smart and industry-standard, smart and low-latency signaling/messaging layer among any kind of needed nodes ( grid-computing, GPU-computing, CLI-treminal, AI/ML-decision making, system-wide consolidated central logging, anything one may need )

Try to setup and use a remote tipc://-transport-class for cross-cluster computing paradigm with any other approach.

Try to setup and use an M:N-redundant Strategy Trading, operated accross a mix of tcp:// + tipc:// + norm:// + vmci:// transport-classes, used so as to interconnect ( A x M + N x B )-nodes' exo-systems.

Try to setup a system, that asks MetaTrader to do some work from outside, without this technology ( webrequest() is not ready for any "qestions from outside", is it? )

Q : "What are the pros and cons compared to the webrequest() function?"

Feel free to read about this in StackOverflow answers.

Question:

I want an Expert Advisor to open an Trade triggerd by a Telegram-Message.

I succesfully set up an Hello-World application using MQ4 as Server and Python/Telegram-bot as Client. When the Telegram-Bot recieves a Message, he will send a request to MQ4 and gets a simple response without executing a trade.

Running Code below.

#   Hello World client in Python
#   Connects REQ socket to tcp://localhost:5555

import zmq
context = zmq.Context()

#  Socket to talk to server
print("Connecting to trading server…")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
print("Connecting to trading server succeed")


#################################################################################
# Use your own values from my.telegram.org
api_id = ######
api_hash = '#####'
bot_token = '#####'
#################################################################################

from telethon import TelegramClient, events

client = TelegramClient('anon', api_id, api_hash)

@client.on(events.NewMessage)
async def my_event_handler(event):


    if "Ascending" in event.raw_text:

        if "AUDUSD" in event.raw_text:
            await event.reply("AUDUSD sell")

            #  Do 1 request, waiting for a response
            for request in range(1):
                print("Telegram: AUDUSD sell execution requested %s …" % request)
                socket.send(b"AUDUSD Sell execute")
                #Send 2 variables (Ordertype // Symbol)

                #  Get the reply. -> Not neccesary for final application
                #  Apülication just needs to send 2 Varianles to MQ4 and trigger the open_order()
                message = socket.recv()
                print("Received reply %s [ %s ]" % (request, message))



client.start()
client.run_until_disconnected()

// Hello World server in MQ4

#include <Zmq/Zmq.mqh>

//+------------------------------------------------------------------+
 void OnTick()
  {
   Context context("helloworld");
   Socket socket(context,ZMQ_REP);

   socket.bind("tcp://*:5555");

   while(!IsStopped())
     {
      ZmqMsg request;

      // Wait for next request from client

      // MetaTrader note: this will block the script thread
      // and if you try to terminate this script, MetaTrader
      // will hang (and crash if you force closing it)

      socket.recv(request);
      Print("Receive: AUDUSD Sell execute");

      Sleep(1000);

      ZmqMsg reply("Trade was executed");
      // Send reply back to client
      socket.send(reply);
      Print("Feedback: Trade was executed");
     }
  }
//+------------------------------------------------------------------+

Now I want to send 2 variables from Python to MQ4. 1. Ordertype: buy/sell 2. Symbol: EURUSD, AUDUSD,...

Send "Sell" if message contains "Ascending" - Send "Buy" if message contains "Descending"

Send "AUDUSD" if message contains "AUDUSD",...

To do so I found a Library from Darwinex and want to combine it (interpretation of message, sending value as an array) with my already functioning telegram-bot.


For testing I wanted to try the example-code from Darwinex by itself.

I found the Code v2.0.1:

Python: https://github.com/darwinex/DarwinexLabs/blob/master/tools/dwx_zeromq_connector/v2.0.1/Python/DWX_ZeroMQ_Connector_v2_0_1_RC8.py

MQ4: (Note: This Library code may replace the whole MQ4 code above in final app.) https://github.com/darwinex/DarwinexLabs/blob/master/tools/dwx_zeromq_connector/v2.0.1/MQL4/DWX_ZeroMQ_Server_v2.0.1_RC8.mq4

When I copy the Code without changing I get an error in Python:

NameError: name '_zmq' is not defined

After running: _zmq._DWX_ZeroMQ_Connector() - in the Kernel of Spyder.

What can I do to fix that error?


In the final state I want to run the Python-Code and the Expert Advisor on the same Windows Server 2012 R2.

Is it enough if I run the .py file in the powershell from the server or should I host the file with the Webside?

I expect to get the whole system/examplecode running on my VPS or Webside-Host-Server and get an testing environment for further coding action, but currenty I cant get the Library Code in Python to run properly.

Also the MT4 ceeps crashing with the current code - but should be fixed if I combine my application with the Library-Codeexample.

(running everything on my local PC with WIN 10).


Answer:

Q : I think it is a connection-problem between MT4 and Python.

Without a fully reproducible MCVE-code this is undecideable.

Having used a ZeroMQ-based bidirectional signalling/messaging between a QuantFX in python and trading ecosystem MetaTrader 4 Terminal implemented in MQL4, there is a positive experience of using this architecture.

Details decide.


The Best Next Step :

Start with a plain PUSH/PULL archetype python-PUSH-es, MQL4-script-PULL-s, preferably using tcp:// transport-class ( win platforms need not be ready to use an even simpler, protocol-less, ipc:// transport-class.

Once you have posack'd this trivial step, move forwards.

Q : How do I need to setup my Server to get a connection betwen those two - since it should be the same as on my local PC?

It is normal to use ZeroMQ on the same localhost during prototyping, so you may test and debug the integration. For details on ZeroMQ, feel free to read all details in other posts.

Q : Is it enough if I run the .py file in the powershell from the server or should I host the file with the Webside I already have and use that as "Python-Server"?

Yes, in case the .py file was designed that way. No code, no advice. That simple.


Possible issues :

Versions - ZeroMQ, since 2.11.x till the recent 4.3.+, has made a lot of changes Installation DLL-details matter.

MQL4 has similarly gone through many changes ( string ceased to be a string and become struct to name a biggest impacting one ), so start with simple scenarios and integrate the target architecture in steps / phases with due testing whether the completed phases work as expected.

Question:

I have a quite weird issue related to ZeroMQ messaging between MetaTrader 4 and a python backend.

The below shown code creates three strings which should be forwarded to a ZeroMQ PUSH socket. However, it only sends the live_trades string via the socket. Both the account_info and the historical_trades strings are not sent.

While debugging the weirdness only increased:

  • When I print the historical_trades string, it is shown in the MT4 expert journal, but not in the MT4 log editor
  • When I add Print( historical_trades ) before or after the pushSocket.send(StringFormat("%s", historical_trades, true)); it sends the string.

Possible issues which I actually exclude:

  • The PUSH socket is too slow: I decreased the timer to 10 seconds and more, same result
  • MT4 itself is too slow to create the strings: It prints them correctly, so it is fast enough
  • The ZeroMQ SNDHWM ( HighWaterMark ) blocks the EA/socket from sending multiple strings in no time: Already increased it to 100 which should be more than sufficient

Any other ideas I cannot verify with debugging:

  • The MT4 Print() function does some other stuff behind the scenes which enables the EA to send the string?
  • Maybe there is a maximum string size in MT4 which blocks the string creation loop (my string is around 35.000 characters long). But then why does it work when I just Print() the string once within the onTimer() function?...
  • Any chance that the account related information is not accessible on weekends?

Or it is just my code which might be buggy..

Any ideas much appreciated!

int OnInit()
  {
//---

   EventSetTimer(2);     // Set Second Timer as push intervall

   context.setBlocky(false);

   // Send data to PULL_PORT that consumer is listening to.
   Print("Connecting MT4 Dashex Feeder to Dashboard on Port " + IntegerToString(PUSH_PORT) + "..");
   pushSocket.connect(StringFormat("%s://%s:%d", ZEROMQ_PROTOCOL, HOSTNAME, PUSH_PORT));

   pushSocket.setSendHighWaterMark(100);
   pushSocket.setLinger(0); 

//---
   return(INIT_SUCCEEDED);
  }


//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
{
//---

   Print("Disconnecting MT4 Dashex Feeder on Port " + IntegerToString(PUSH_PORT) + "..");
   pushSocket.disconnect(StringFormat("%s://%s:%d", ZEROMQ_PROTOCOL, HOSTNAME, PUSH_PORT));

   // Shutdown ZeroMQ Context
   context.shutdown();
   context.destroy(0);

   EventKillTimer();
}


//---

//+------------------------------------------------------------------+
//| Expert timer function                                             |
//+------------------------------------------------------------------+
void OnTimer()
{

   /*
      1.) Account information
   */   

      string account_info = "";   
      account_info = account_info +

         "account_info|" +
         version + "|" +
         DID + "|" +
         IntegerToString(AccountNumber()) + "|" +
         AccountInfoString(ACCOUNT_COMPANY) + "|" +
         IntegerToString(AccountInfoInteger(ACCOUNT_LEVERAGE)) + "|" +
         DoubleToString(AccountInfoDouble(ACCOUNT_BALANCE),2) + "|" +
         DoubleToString(AccountInfoDouble(ACCOUNT_PROFIT),2) + "|" +
         DoubleToString(AccountInfoDouble(ACCOUNT_EQUITY),2) + "|" +
         DoubleToString(AccountInfoDouble(ACCOUNT_MARGIN),2) + "|" +
         DoubleToString(AccountInfoDouble(ACCOUNT_MARGIN_FREE),2) + "|" +
         DoubleToString(AccountInfoDouble(ACCOUNT_MARGIN_LEVEL),2) + "|" +
         AccountInfoString(ACCOUNT_CURRENCY) + "|" +
         IntegerToString(IsDemo()) + "|" +

         pushSocket.send(StringFormat("%s", account_info, true));
         Print("Pushing Account Information To Dashex.Finance Dashboard For Account No. " + IntegerToString(AccountNumber()));


//+------------------------------------------------------------------+
//+------------------------------------------------------------------+         

   /*
      2.) Pending and running trades
   */

      int live_orders = OrdersTotal();
      string live_trades = "";

      for(int i=live_orders; i >= 0; i--)
      {
         if(OrderSelect(i,SELECT_BY_POS)==false) continue;

           live_trades = live_trades +

           "live_trades|" +
           version + "|" +
           DID + "|" +
           IntegerToString(AccountNumber()) + "|" +
           IntegerToString(OrderTicket()) + "|" +
           TimeToString(OrderOpenTime(), TIME_DATE|TIME_SECONDS) + "|" +
           TimeToString(OrderCloseTime(), TIME_DATE|TIME_SECONDS) + "|" +
           IntegerToString(OrderType()) + "|" +
           DoubleToString(OrderLots(),2) + "|" +
           OrderSymbol() + "|" +
           DoubleToString(OrderOpenPrice(),5) + "|" +
           DoubleToString(OrderClosePrice(),5) + "|" +
           DoubleToString(OrderStopLoss(),5) + "|" +
           DoubleToString(OrderTakeProfit(),5) + "|" +
           DoubleToString(OrderCommission(),2) + "|" +
           DoubleToString(OrderSwap(),2) + "|" +
           DoubleToString(OrderProfit(),2) + "|" +
           "<" + OrderComment() + ">|";
      }

      pushSocket.send(StringFormat("%s", live_trades, true));
      Print("Pushing Live Trades To Dashex.Finance Dashboard For Account No. " + IntegerToString(AccountNumber()));

//+------------------------------------------------------------------+
//+------------------------------------------------------------------+

   /*
      3.) History Trades      
   */


         int hstTotal = OrdersHistoryTotal();
         string historical_trades = "";

         Print(hstTotal);
         for(int i=hstTotal; i >= 0; i--)
         {

           if(OrderSelect(i,SELECT_BY_POS,MODE_HISTORY)==false) continue;

           historical_trades = historical_trades +

              "historical_trades|" +
              version + "|" +
              DID + "|" +
              IntegerToString(AccountNumber()) + "|" +
              IntegerToString(OrderTicket()) + "|" +
              TimeToString(OrderOpenTime(), TIME_DATE|TIME_SECONDS) + "|" +
              TimeToString(OrderCloseTime(), TIME_DATE|TIME_SECONDS) + "|" +
              IntegerToString(OrderType()) + "|" +
              DoubleToString(OrderLots(),2) + "|" +
              OrderSymbol() + "|" +
              DoubleToString(OrderOpenPrice(),5) + "|" +
              DoubleToString(OrderClosePrice(),5) + "|" +
              DoubleToString(OrderStopLoss(),5) + "|" +
              DoubleToString(OrderTakeProfit(),5) + "|" +
              DoubleToString(OrderCommission(),2) + "|" +
              DoubleToString(OrderSwap(),2) + "|" +
              DoubleToString(OrderProfit(),2) + "|" +
              "<" + OrderComment() + ">|"; 
         }

         pushSocket.send(StringFormat("%s", historical_trades, true));
         Print("Pushing History Trades To Dashex.Finance Dashboard For Account No. " + IntegerToString(AccountNumber()));

         Sleep(1);
}

MetaTrader 4 log.editor:

MetaTrader 4 console:


Answer:

Q : Any chance that the account related information is not accessible on weekends?

This has an easy proof: do a Comment( aStringUnderTEST ); + check all account / Broker-related items.


Q : Maybe there is a maximum string size in MT4 which blocks the string creation loop...?

This is, except a case of an MT4-Build-X.Y.Z release-bug awaiting a BugFIX, very low probability hypothesis.

Yet it has an easy proof: do a loop of growing string-lengths and test until what size the processing keeps working. The indirectly proved size-limit will help you track the root cause, MT4 not being the SPoF here, is it?


Q : The MT4 Print function does some other stuff behind the scenes which enables the EA to send the string?

The MT4 Support ought be contacted if this is to get confirmed or denied. The software is licensed as an as-is product, so do not expect any rapid response or a rocket science to take place if you try to drill down to the bolts and nuts, inside their closed and sealed product.

Question:

I have an EA set in place that loops history trades and builds one large string with trade information. I then send this string every second from MT4 to the python backend using a plain PUSH/PULL pattern.

For whatever reason, the data isn't received on the pull side when the string transferred becomes too long. The backend PULL-socket slices each string and further processes it.

Any chance that the PULL-side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?

Talking about file sizes we are well below 5kb per second.

This is the PULL-socket, which manipulates the data after receiving it:

while True:
    # check 24/7 for available data in the pull socket
    try:
        msg = zmq_socket.recv_string()
        data = msg.split("|")
        print(data)

        # if data is available and msg is account info, handle as follows
        if data[0] == "account_info":
[...]
    except zmq.error.Again:
        print("\nResource timeout.. please try again.")
        sleep(0.000001)

I am a bit curious now since the pull socket seems to not even be able to process a string containing 40 trades with their according information on a single MT4 client - Python connection. I actually planned to set it up to handle more than 5.000 MT4 clients - python backend connections at once.


Answer:

Q : Any chance that the pull side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?

Zero chance.

Sending 640 B each second is definitely no showstopper ( 5kb per second - is nowhere near a performance ceiling... )

The posted problem formulation is otherwise undecidable.

Step 1) POSACK/NACK prove whether a PUSH side accepts the payload for sending error-free.

Step 2) prove the PULL side is not to be blamed - [PUSH.send(640*chr(64+i)) for i in range( 10 )] via a python-2-python tcp://-transport-class solo-channel crossing host-to-host hop, over at least your local physical network ( no VMCI/emulated vLAN, no other localhost colocation )

Step 3) if either steps above got POSACK-ed, your next chances are the ZeroMQ configuration space and/or the MT4-based PUSH-side incompatibility, most probably "hidden" inside a (not mentioned) third party ZeroMQ wrapper used / first-party issues with string handling / processing ( which you must have already read about, as it has been so many times observed and mentioned in the past posts about this trouble with well "hidden" MQL4 internal eco-system changes ).

Anyway, stay tuned. ZeroMQ is a sure bet and a truly horsepower for professional and low-latency designs in distributed-system's domain.

Question:

I am having tough times to grab and print data which is successfully pushed to the wire. I have set up a plain Push-Pull-Architecture with Metatrader 4 acting as producer and Python backend acting as consumer.

When it comes to grabbing and printing the data, I just cannot make it happen.

This is my push instance which works just fine:

Metatrader 4 zeroMQ push instance:

#include <Zmq/Zmq.mqh>

// EA plot settings

extern string PROJECT_NAME = "Dashex.Feeder";
extern string ZEROMQ_PROTOCOL = "tcp";
extern string HOSTNAME = "localhost";
extern int PUSH_PORT = 32225;

extern string t0 = "--- Feeder Parameters ---";
input string DID = "insert your DID here";
extern string t1 = "--- ZeroMQ Configuration ---";
extern bool Publish_MarketData = false;

// ZeroMQ environment //

// CREATE ZeroMQ Context
Context context(PROJECT_NAME);

// CREATE ZMQ_PUSH SOCKET
Socket pushSocket(context, ZMQ_PUSH);

string Publish_Symbols[7] = {
   "EURUSD","GBPUSD","USDJPY","USDCAD","AUDUSD","NZDUSD","USDCHF"
};

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+

int OnInit()
  {
//---

   EventSetTimer(1);     // Set Millisecond Timer to get client socket input

   context.setBlocky(false);

   // Send responses to PULL_PORT that client is listening on.
   Print("[PUSH] Connecting MT4 Server to Socket on Port " + IntegerToString(PUSH_PORT) + "..");
   pushSocket.connect(StringFormat("%s://%s:%d", ZEROMQ_PROTOCOL, HOSTNAME, PUSH_PORT));

   pushSocket.setSendHighWaterMark(1);
   pushSocket.setLinger(0);   

//---
   return(INIT_SUCCEEDED);
  }

//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
{
//---

   Print("[PUSH] Disconnecting MT4 Server from Socket on Port " + IntegerToString(PUSH_PORT) + "..");
   pushSocket.disconnect(StringFormat("%s://%s:%d", ZEROMQ_PROTOCOL, HOSTNAME, PUSH_PORT));

   // Shutdown ZeroMQ Context
   context.shutdown();
   context.destroy(0);

   EventKillTimer();
}
//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTimer()
{
   /*
      Use this OnTimer() function to send market data to consumer.
   */
   if(!IsStopped() && Publish_MarketData == true)
   {
      for(int s = 0; s < ArraySize(Publish_Symbols); s++)
      {

         string _tick = GetBidAsk(Publish_Symbols[s]);
         Print("Sending " + Publish_Symbols[s] + " " + _tick + " to PUSH Socket");
         ZmqMsg reply(StringFormat("%s %s", Publish_Symbols[s], _tick));
         pushSocket.send(reply, true);
      }
   }
}
//+------------------------------------------------------------------+

string GetBidAsk(string symbol) {

   MqlTick last_tick;

   if(SymbolInfoTick(symbol,last_tick))
   {
       return(StringFormat("%f;%f", last_tick.bid, last_tick.ask));
   }

   // Default
   return "";
}

How data is pushed to the wire (array of strings, snippet from above code):

         string _tick = GetBidAsk(Publish_Symbols[s]);
         Print("Sending " + Publish_Symbols[s] + " " + _tick + " to PUSH Socket");
         ZmqMsg reply(StringFormat("%s %s", Publish_Symbols[s], _tick));
         pushSocket.send(reply, true);

Python side pull client which doesn't grab/print any data from the wire:

import zmq
import time
from time import sleep

context = zmq.Context()
zmq_socket = context.socket(zmq.PULL)
zmq_socket.bind("tcp://*:32220")
time.sleep(1)

while True:

    def pull_MT4():

        try:
            msg = zmq_socket.recv_string()
            print(msg)
            return msg
        except zmq.error.Again:
            print("\nResource timeout.. please try again.")
            sleep(0.000001)

        return None

Console output without printed data :-(


Answer:

Inside while True you only define function pull_MT4 again and again but you never execute this function - you never use pull_MT4() to run it.

You should use this code directly in loop (without define function and without return)

while True:
    try:
        msg = zmq_socket.recv_string()
        print(msg)
    except zmq.error.Again:
        print("\nResource timeout.. please try again.")
        sleep(0.000001)

or you should define function before loop and execute it inside loop

def pull_MT4(): # define function
    try:
        msg = zmq_socket.recv_string()
        print(msg)
        return msg
    except zmq.error.Again:
        print("\nResource timeout.. please try again.")
        sleep(0.000001)
    return None

while True:
    pull_MT4() # execute function