QUANTCONNECT COMMUNITY
At the moment, I have 2CPUs | 8GB RAM for backtesting. I am trying to develop an ML based algo off of my strategies. Does anyone have advice on how to size the nodes for the backtesting of the ML? Using 3 symbols, 25 -35 features, minute data, and a training sample of ~TBD (would like 2000). I am running a stripped down test with only 250 samples in the training and is taking forever. If it makes a difference I am using xgboost and a neural net ML approach.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
QuantConnect™ 2024. All Rights Reserved
|
|
Share New Research 
Louis Szeto
Hi Michael
IMHO, the speed of training is less important if you’re using bar-width sparser than 15-minute bar and not very frequent recalibration interval. Most NN and gradient-boost models have measures to advance their speed. I’ll focus on calculating the required RAM capacity.
First one would be the capacity required for the model parameters. Say you’re building a Kara’s model, you may build your model’s infra-structure in a research notebook and call the model.summary() method to checkout the total parameters needed to be calibrate. Then multiply by 16 (most QC data were using 16-byte decimal) to get the estimated bytes needed.
Then estimate the data batch size. 25-35 features per 3 symbols for 2000 datapoints will need around 2.5MB to 3.5MB (for 16-byte data). Add those numbers up and you’ll get the minimum required memory and guess the backtest/research node you’d need.
But if speed is important, then it is better to do things in local environment as GPU can be supported. Take a look this thread to see how to set up.
Best
Louis
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Michael Dehring
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!