Unable to Find Algorithm Memory Leak by Brian Christopher

Brian Christopher INVESTOR

| Reward Discussion

Forum,

I'm looking for assistance improving efficiency and/or finding the memory leak in this algorithm. Note the `RAM` in the chart below is consistently trending upward with major spikes at various times.

The algorithm itself is a relatively simple long-only strategy that buys at potential market bottoms. It uses `sklearn` to fit a Gaussian Mixture, then `scipy.stats` to sample from a distribution. The only thing saved between runs is the buy list which is a `np.ndarray`. The algo is scheduled to run twice weekly, resolution is hourly, history is `252*6.5` hours.

Thanks guys.

Update Backtest

Brian Christopher

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Derek Tishler

16.1k Pro ,

Here is a backtest with some changes I made to get you started on optimizing.

I think part of the linear trend is the accumulation of rolling metrics, orders, trades, and more. I imagine Lean has to do some heavy lifting to offer us all the info provided in Overview, Trades, etc.

I switched to list comprehension on the main compute code, instead of the existing for loop, using something like:
results = [eat(cheese) for cheese in wheel if cheese.owner == 'Me'] #yummy

I traded a bit of ram for some cpu. The large, redundant use of History was slowing things down a bit and would be hard to accomplish in live without a timeout or delay. I switched this to a dictionary of length limited DataFrames to reduce how much history needs to be requested at each rebalanced. I also think you can increase speed by adjusting your hyperparameters such as tol, max_iter, and n_itit to limit time spent waiting for each model's loss to converge or for the training to give up.

I added an elapsed time tracker to the chart to ensure things are not blowing up there. The original algo had times between 0 and 20, averaging over 10+ sec. I also switched the plotting of ram to occur every day just in case some kind of post compute memory use was inflating the Ram chart.

Brian Christopher INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

Community Ambassadors

Earn free QuantConnect cloud credits by helping others, each week we airdrop credits to the most active community members.

232,035 Quants.

Online Now

Stay Connected