Back

[Question] Backtesting performance

Hi there! I'm performing experiments with LEAN backtesting and high-performance computing.
Master software is starting hundreds of LEAN launcher instances in my setup on hi-end HPC configurations. Algorithm result json storing and file logging are turned off to evade unnecessary load and file write conflicts.


The problem is all about very poor backtesting performance. Typical algorithm configuration with dozen of rolling windows and indicators on minute resolution and 15-minute consolidation (where all the windows are updated) on 10-years duration takes up to 1 minute to backtest.
Is it possible to dramatically increase backtesting performance?

Algorithm is all about this:

var consolidator =
new QuoteBarConsolidator(TimeSpan.FromMinutes(15));
consolidator.DataConsolidated += ConsolidatorOnDataConsolidated;

_rsi = new RelativeStrengthIndex(14);
_cci = new CommodityChannelIndex(30);
_adx = new AverageDirectionalIndex("ADX", 14);
_stoch = new Stochastic(14, 14, 3);
_williamsPercentR = new WilliamsPercentR(14);
_trix = new Trix(15);
_aroon = new AroonOscillator(14, 14);
_adxr = new AverageDirectionalMovementIndexRating(14);
_ultimateOscillator = new UltimateOscillator(7, 14, 28);

RegisterIndicator(_symbol, _rsi, consolidator);
RegisterIndicator(_symbol, _cci, consolidator);
RegisterIndicator(_symbol, _adx, consolidator);
RegisterIndicator(_symbol, _stoch, consolidator);
RegisterIndicator(_symbol, _williamsPercentR, consolidator);
RegisterIndicator(_symbol, _trix, consolidator);
RegisterIndicator(_symbol, _aroon, consolidator);
RegisterIndicator(_symbol, _adxr, consolidator);
RegisterIndicator(_symbol, _ultimateOscillator, consolidator);

And custom windows populating in consolidator callback.

Hardware: Amazon EC2 c5d.18xlarge instance:

72 3.0 GHz vCPUs

144 Gib RAM

2 x 900 GB NVMe SSD

Thank you for any suggestions in advance!There are some release run profiling results:

Update Backtest







Hi what data are you running? Tick, minute, universe, fx?
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


That's

AddForex(_symbol, Resolution.Minute, Market.Oanda);

On

SetStartDate(2010, 01, 03);
SetEndDate(2018, 01, 01);

0

Interesting results - just wanted to add my own experiences to this, also on Minute data. I have run with two different set-ups and had differing results:

  • Setup #1: Similar to yours - I have a master that spawns LEAN slave processes. Each process performs one backtest with one alpha on one underlying. This is the most efficient way I've found of completing backtests. I've tried this on a Google Cloud highcpu instance with 16 CPU cores and been able to run about 10 of them in parallel before exhausting the CPU.
  • Setup #2: Here I'm trying one LEAN process - 20 securities, 9 Alphas (each with one consolidator and 1-2 indicators, producing 100-1000 insights per year per alpha per underlying). This is running significantly slower and is only able to use ~30% of my (local) i7-4710MQ. From testing, I am seeing that 15 years of backtesting would take something like 30-45 hours to complete. I have transferred this to the QC web terminal, and initially it runs quite fast but the performace drops off substantially after a few hours. On a 4 CPU core GCloud instance this doesn't seem to be particularly faster either, as it simply cannot use a substantial amount of the processing power (only about 50%).
When profiling setup #2, I can see that roughly 80% of CPU time is spent on sleeping and sychronisation. I'm wondering whether there is scope from the QC team to work on supporting large, multi-asset Framework algorithms like these? I think as people adopt the Framework approach, many will try to add more Alphas and more underlyings and so might run into these bottlenecking issues as well.
0

We're investing alot of energy making it run faster. Feel free to check out

the progress in the Python speed project.



https://github.com/QuantConnect/Lean/projects
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed

 
";