Back

Unable to Find Algorithm Memory Leak

Forum,

I'm looking for assistance improving efficiency and/or finding the memory leak in this algorithm. Note the `RAM` in the chart below is consistently trending upward with major spikes at various times. 

The algorithm itself is a relatively simple long-only strategy that buys at potential market bottoms. It uses `sklearn` to fit a Gaussian Mixture, then `scipy.stats` to sample from a distribution. The only thing saved between runs is the buy list which is a `np.ndarray`. The algo is scheduled to run twice weekly, resolution is hourly, history is `252*6.5` hours. 

Thanks guys.

Update Backtest








Here is a backtest with some changes I made to get you started on optimizing. 

I think part of the linear trend is the accumulation of rolling metrics, orders, trades, and more. I imagine Lean has to do some heavy lifting to offer us all the info provided in Overview, Trades, etc.

I switched to list comprehension on the main compute code, instead of the existing for loop, using something like:
results = [eat(cheese) for cheese in wheel if cheese.owner == 'Me'] #yummy

I traded a bit of ram for some cpu. The large, redundant use of History was slowing things down a bit and would be hard to accomplish in live without a timeout or delay. I switched this to a dictionary of length limited DataFrames to reduce how much history needs to be requested at each rebalanced. I also think you can increase speed by adjusting your hyperparameters such as tol, max_iter, and n_itit to limit time spent waiting for each model's loss to converge or for the training to give up.

I added an elapsed time tracker to the chart to ensure things are not blowing up there. The original algo had times between 0 and 20, averaging over 10+ sec. I also switched the plotting of ram to occur every day just in case some kind of post compute memory use was inflating the Ram chart.

0


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed