Share New Research 
Hello
I currently have an algo that doesn't do that much stuff (IMO), yet it's getting killed randomly by out of memory events during back tests. It would be rather bad if it happened during live deployment. Given the non-deterministic nature of GC, chances are it sometimes will. Is there a way to detect these events ahead of time? Such limits are (assuming one runs 64 bit executables) on a dedicated machine typically softer and cause incremental slow down way ahead of running out (since virtual memory can be paged to slower SSD/HDD).
I've deliberately coded in a non-memory optimal manner (because I didn't expected memory to be an issue). I could rewrite my own algo to use zero transient allocations code patterns, but then there remains the issue of LeanEngine happily allocating temporary objects (from the very little I've looked at the code so far, it loves using Linq). Admittedly, with proper GC, these objects should die in gen 0/1 collections before causing loss of performance in gen 2, however, the greater system stress the greater chance of a build up in allocations before next GC cycle.
If anyone has any QC specific guidelines of memory usage, please share. Currently I'm somewhat skeptical of being able to build algorithms that regularly survive longer backtests with more than a few indicators/data series. Will certainly be looking more into the matter though.
Petter Hansson
My hypothesis is problem is constrained to minute tick backtests (haven't seen it happen on daily or second resolution yet). I'll work around it for now.
Long term I guess some kind of early warning of impending memory shutdown would be nice but that depends a lot on what Mono can do ofc.
Jared Broad
Thank you Petter for the information; its a hard problem for sure. Mono's GC simply isn't as good as .NET; but there aren't many good/easy containerization solutions for Windows environments. For now we've chosen security & the stability of linux over the memory issue. The good news is Docker has announced support for Windows containers which will let us backtest in .NET. This should be possible (out of beta) in the next 6-12 months.
The good news is LEAN in Windows uses almost no RAM... and even in Linux barely uses RAM most of the time. Its just some rare occassions where it seems to really hog! You can use GC.GetTotalBytes() to get an idea of your usage -- this is how we control and kill the backtests in LEAN.
We'll consider extending the base-level RAM for clients; mostly the RAM limits are in place to prevent silly stuff like adding all data to a list... storing everything as strings etc. If you can't do simple backtests without hitting it its definitely a problem we'll solve ASAP.
PS: If you're a memory expert we could really use a hand with this! Its hard to repeat / diagnose.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Petter Hansson
Yes, this behavior might be Mono specific, but it's difficult to know. The most recent Mono GC sgen isn't that much worse than .NET, and I researched this matter rather extensively on a previous work assignment.
GC behavior in practice is highly non-linear, especially during stress, which unfortunately (to use descriptive terms) means it can take a break and assume that it's fine to let memory usage grow beyond some fixed limit unknown to it. I'm somewhat concerned GC.GetTotalMemory(true) will kill your app even when if it's due to temporary objects piling up (before, or even after collection depending on how memory is released to the OS). The wording of the following implies it might decide not to wait:
"If the forceFullCollection parameter is true, this method waits a short interval before returning while the system collects garbage and finalizes objects. The duration of the interval is an internally specified limit determined by the number of garbage collection cycles completed and the change in the amount of memory recovered between cycles. The garbage collector does not guarantee that all inaccessible memory is collected."
I assume you've setenv MONO_GC_PARAMS max-heap-size=4G?
I'll not be available in August but I'll try to look at memory allocations in Lean in September provided I'm still in the loop at that time.
Petter Hansson
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!