Discussion Forum - QuantConnect.com

Erez Tison INVESTOR

| Reward Discussion

Hey there,

I'm trying to fetch indicator values on a cloud research notebook for about 3000+ different combinations of symbol and time (total of 90 stocks), the indicator is RSI for example with 100 periods.

The thing is that after a few minutes an OutOfMemoryException shows up, running on an R-MICRO node (2GB RAM). I tried upgrading the research node to R1-4 (4GB RAM) but got the same problem.

Few questions if I may:

Is there a way to view a research node's realtime resource consumption (CPU, memory) ?
Is there a list of limitations of the research nodes somewhere?
I noticed the “Datasets Subscriptions” in the pricing page, what exactly does it allows? would I be able to download a 1-minute data for many stocks and research locally?

Any advise on how to workaround this limitation will be appreciated, partitioning is considered. Below is the full exception.

Thank you!

OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.AddWithResize(T item)
at QuantConnect.Research.QuantBook.<>c__DisplayClass25_0.<Indicator>b__3(Object s, IndicatorDataPoint e) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Research/QuantBook.cs:line 722
at QuantConnect.Indicators.IndicatorBase`1.Update(IBaseData input) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Indicators/IndicatorBase.cs:line 93
at QuantConnect.Indicators.IndicatorBase`1.Update(DateTime time, Decimal value) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Indicators/IndicatorBase.cs:line 134
at QuantConnect.Research.QuantBook.<>c__DisplayClass25_0.<Indicator>b__5(BaseData bar) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Research/QuantBook.cs:line 731
at QuantConnect.Data.SliceExtensions.PushThrough(IEnumerable`1 slices, Action`1 handler) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Common/Data/SliceExtensions.cs:line 222
at QuantConnect.Research.QuantBook.Indicator(IndicatorBase`1 indicator, IEnumerable`1 history, Func`2 selector) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Research/QuantBook.cs:line 728
at QuantConnect.Research.QuantBook.Indicator(IndicatorBase`1 indicator, Symbol symbol, DateTime start, DateTime end, Nullable`1 resolution, Func`2 selector) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Research/QuantBook.cs:line 549

Update Backtest

Erez Tison

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Varad Kabade

31.1k Pro ,

Hi Erez Tison,
OutOfMemoryException occurs when you have more data in the memory than is available (4 gb). This can be avoided by optimizing the code; please share the attached backtest for more assistance

Is there a way to view a research node's realtime resource consumption (CPU, memory) ?

No currently, we cannot view the memory usage

Is there a list of limitations of the research nodes somewhere?

In the pricing page, you can find the research node limits for Free,Quant Researcher,Team,Trading Firm and Institution plan

I noticed the “Datasets Subscriptions” in the pricing page, what exactly does it allows? would I be able to download a 1-minute data for many stocks and research locally?

Yes, with a data subscription, you can use the data for research and develop algorithms locally at all the available resolutions. You can add the Security Master dataset to have all the US Equity data, or can download individual files from the Data Explorer with the CLI.Best,
Varad Kabade

Erez Tison

1k Pro ,

Thank you Varad kabade for the prompt answer.
Assuming the research node memory limit was doubled when I upgraded to from R-MICRO node (2GB RAM) to R1-4 (4GB RAM), and the problem persist, is there another per-kernel limitation or something similar that is not enlarged in an upgrade?

The code is really simple, just qb.indicator for RSI with 110 periods, I did noticed that lesser periods may avoid this problem.

Thank you.

Varad Kabade

31.1k Pro ,

Hi Erez,
We recommend trying to perform your analysis/research in batch (some stocks at a time) and storing the result in ObjectStore, and reviewing them as needed. This would make sure that we don't exhaust our working memory. Would you please share a dedicated backtest containing the research notebook for further assistance? Refer to the following thread regarding storing and retrieving data to/from the ObjectStore.
Best,
Varad Kabade

Erez Tison

1k Pro ,

Thank you Varad, will try partitioning and ObjectStore.

Best, Erez.

Erez Tison INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240900 Quants.

Online Now

Stay Connected

OutOfMemoryException error on research notebook

Started By:

Join QuantConnect for Free Sign Up

Platform

SIGN IN

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240900 Quants.

Online Now

Stay Connected

OutOfMemoryException error on research notebook

Started By:

Join QuantConnect for Free Sign Up