Reversed array with MQL4

arraysetasseries mql5
mql4 array
mql4 array resize
mql4 find in array
mql4 dynamic array
mql4 array average
mql4 clear array
mql4 sort array

With MetaTrader Terminal ( MQL4 ), I try to have a reversed array, that I append ( prepend ) items to.

So, on every tick, myArray[0] becomes the 'newest' value, and the previous value shifts to myArray[1] and so on.

But its harder then it sounds.

I tried like this ->

       double myArray        = [];                        // GLOBAL Dynamic array
extern int    maxArrayLength = 50;                        // EXTERN iterable

// -----------------------------------------------------------------------
bool   prependToReversedDoubleArray( double& theArray[], double value, int maxLength ) {

       int size = ArraySize( theArray );                 // LOCAL size
       ArraySetAsSeries(     theArray, false );          // Normalize the array ( left to right )
       ArrayResize(          theArray, size + 1 );       // Extend array length

       Alert( "test = ", size );

       theArray[size] = value;                           // Insert the new value
       ArraySetAsSeries(     theArray, true );           // Reverse the array again
       if ( ArraySize(       theArray ) > maxLength ) {
            ArrayResize(     theArray,    maxLength );
       }
       return( true );
}

prependToReversedDoubleArray( myArray, 0.1234, maxArrayLength );
for( int i = arraylength - 1; i >= 1; i-- ) {
    value[i] = value[i-1];
    }
value[0] = newValue;

Array Functions / ArrayReverse, Array Functions / ArrayReverse - Reference on algorithmic/automated trading language for MetaTrader 5. void& array[], // array of any type uint start=0, // index to start reversing the array from uint count=WHOLE_ARRAY // number of  Group of Functions for Working with Arrays. Arrays are allowed to be maximum four-dimensional. Each dimension is indexed from 0 to dimension_size-1.In a particular case of a one-dimensional array of 50 elements, calling of the first element will appear as array[0], of the last one - as array[49].


Intro:

Fortunately a default MQL4 instrumentation for TimeSeries organisation will not work on this scenario.

Why?

MQL4 TimeSeries ( reversed ) Arrays do get system-driven event-locked cell-index re-shuffling only on a current TimeFrame's aNewBarEVENT, not based on just every anFxQuoteArrivalEVENT ( as was asked in the O/P to shift / update [0] per each tick arrival " ..., on every tick, " ).


How to make it work somehow?

A trivial for(){ shift 'em all / store new} kind of loops, proposed earlier, seems on a first glimpse, as a simple can-do hack.

The danger is a devil is hidden in details.

After some 100.000+ quotes, the array(s) grow to sizes a single memory-page will not hold the whole array + processing times for a dumb cell-shifting grows ( linearly ) in O(1), but to such scales, which start to destroy an ability to be still as fast as being finally able to wait a few ms / us for next FOREX market events' arrival in a non-blocking mode, thus the MetaTrader Terminal inner architecture loses an ability to hold an illusion of a false-synchronicity with external events.

ArrayResize() is another hidden devil out there.

In other words, such code will start "missing" events ( will drop data ( which it will never see on arrival, as still shuffling data in the cell-shifting loop ) ).


How to make it work fast & smart?

0 ) Avoid memory-page swaps - stay In-RAM.

1 ) Avoid any blocking step.

2 ) Avoid a dumb cell-shuffling of any kind - alike value[i] = value[i-1];.

3 ) Avoid any ArrayResize() on the fly.

Solution leads to a proxy in a form of a circular-buffer architecture with a distributed ( the only possible non-blocking help to an MT4 MQL4 code-execution with a rigid, user non-controllable, thread architecture )

This way MQL4 code can contain a lightweight proxy-object ( internally a local, cache-alike managed ring buffer ) which can also seamlessly access literally limitless amounts of cell-data, stored and practically maintained in a remote process / a fast computing grid.

This is both non-blocking (ever) and fast and smart and limitless ( if your algo-trading needs that ) .

ArraySetAsSeries - Array Functions - MQL4 Reference, The function sets the AS_SERIES flag to a selected object of a dynamic array, and elements will be indexed bool flag // true denotes reverse order of indexing The AS_SERIES flag can't be set for multi-dimensional arrays or static arrays (arrays, whose size in square brackets is preset already on the compilation stage). Indexing in timeseries differs from a common array in that the elements of timeseries are indexed from the end towards the beginning (from the newest to oldest data).

thanks for all the good intel! I learned a lot from the explanations :)

The problem I had was, how to append values to the 'beginning' of a reversed array ( write to array[0] and shift the rest up ).It still not non-blocking and probably not the fastest way, but it works for now.

Here is the solution, it also takes a 'maxLength' value, that keeps the array size as desired :) :

int prependToReversedDoubleArray(  double &theArray[],
                                   double  value,
                                   int     maxLength
                                   )
{   int newSize = ArraySize( theArray ) + 1;
    ArraySetAsSeries( theArray, false );     // Normalize the array (left to right)
    ArrayResize(      theArray, newSize );   // Extend array length

    theArray[newSize-1] = value;             // Insert the new value

    ArraySetAsSeries( theArray, true );      // Reverse the array again

    if (  maxLength > 0
       && newSize   > maxLength )            // Check if the max length is reached
    {     newSize   = maxLength;
          ArrayResize( theArray, maxLength );
    }
    return( newSize );
}

Indexing Direction in Arrays, Buffers and , Timeseries are arrays with reverse indexing, i.e. the first element of a timeseries is in the extreme right position, and the last element is in the extreme left position​  Timeseries are arrays with reverse indexing, i.e. the first element of a timeseries is in the extreme right position, and the last element is in the extreme left position. Timeseries being used for storing history price data and contain the time information, we can say that the newest data are placed in the extreme right position of the timeseries, while the oldest data are in the extreme left position.

This can be done for optimal performance by creating your own dynamic Array class based on a linked list of pointers. You can define a custom operator [] to access its elements like you do with a normal array. This class can implement a Prepend() method that would be very fast as it would just have to perform a few operations with pointers and just one memory allocation.

However, such a class is not a simple thing to code, so depending on your goals, it might not be the best solution to your case.

ArrayBsearch - Array Functions - MQL4 Reference, Searches for a specified value in a multidimensional numeric array sorted in the ascending order. MODE_DESCEND searching in backward direction. All the subsequent calls of ArrayResize do not lead to physical reallocation of memory, but only change the size of the first array dimension within the reserved memory. It should be remembered that the third parameter will be used only during physical memory allocation.

Close - Predefined Variables - MQL4 Reference, Series array that contains close prices for each bar of the current chart. Series array elements are indexed in the reverse order, i.e., from the last one to the first  Array-timeseries is an array with a predefined name (Open, Close, High, Low, Volume or Time), the elements of which contain values of corresponding characteristics of historic bars. Data contained in arrays-timseries carry a very important information and are widely used in programming in MQL4.

Low - Predefined Variables - MQL4 Reference, Series array that contains the lowest prices of each bar of the current chart. Series array elements are indexed in the reverse order, i.e., from the last one to the  MQL4 TimeSeries ( reversed ) Arrays do get system-driven event-locked cell-index re-shuffling only on a current TimeFrame's aNewBarEVENT, not based on just every anFxQuoteArrivalEVENT ( as was asked in the O/P to shift / update [0] per each tick arrival " , on every tick, ").

Reversed array with MQL4 - low-latency, With MetaTrader Terminal ( MQL4 ), I try to have a reversed array, that I append ( prepend ) items to. So, on every tick, myArray[0] becomes the 'newest' value,  If the array is a series array, then the end of the array is on the left hand side, because series arrays are indexed right to left. Here is a code snippet that incorporates all that has been discussed on this thread. It will resize any array, series OR non-series, up to a maximum size.

Comments
  • And what is The Question, Kev?
  • [0] element always will be the newest.
  • [0] element may be the newest, the problem is, the reversed array will not help in given task ( ref. for details below )
  • While this is possible, one may kindly realise it is computationally very inefficient on growing scales of the ArraySize(). High performing tasks shall not block soft-real-time execution flow with uncontrollable mem-swaps and dumb cell-shifting per each tick arrival ( cadence in the order of [ms] ) -- where both of these fatal drawbacks are out of your control and fortunately present in the proposed algorithmisation. A better approach exists.
  • @user3666197.. I can already notice that when running multiple custom indicators in an EA, its not getting much faster.. So any info about efficient looping and calculating is more then welcome :)
  • Kev, the code-execution of Technical Indicator-s got the hardest hit from the New-MQL4.56789. It shares one single thread !!! ( yes, ALL indicators are being executed under a limit of one shared thread ...!!! ). One can imagine, what that makes with deep-recursive algorithms ( some of which were almost killing the old-MT4-model of code-execution ... ) .... so yes, indicators can & DO BLOCK main Expert Advisor code-execution ...
  • Ai, thats not good. I just posted another question (stackoverflow.com/questions/34355907/…) and I think its gonna be impossible.. I found some code that made it possible to use NodeJS as 'calculating' engine and communicating with MQL4.. You probably could clone each NodeJS instance and thus use multi-core. Have you ever tried something like that? I expect it to give a lot of troubles, but maybe there are more guys doing it?
  • Yes, we use MQL4 code, interconnected with many remote external actors ( while the motivation is not uniform, the technology is, which increases the ROI spent on learning curve and mastering the low-latency aspects of the distributed infrastructure ).
  • Is this viable for say...10 arrays? I need to have several arrays for this, as I am trying to translate a complex pine script (tradingview) script to MQL4. Over there, every variable just magically works as a series, but it seems much more difficult in mql4.
  • This would make it quite simple, as long as it doesn't lag out when i'm constantly updating several arrays on every new candle