Hello all,
I ran a notebook yesterday with no indication whatsoever of it being strenuous on my research node. However, when I ran it today, the notebook shut down, because the research node ran out of memory. This is not the first time this has happened, and I have tried to optimize memory efficiency in various ways, at the cost of my research quality. Is there anything I can do, beyond purchasing more research nodes in order to avoid this issue?
Mia Alissi
Hello Zachary,
It seems like you're encountering memory issues with your research node. This can happen when the data or computations you're working with exceed the memory capacity of your research node. Here are a few suggestions based on the information from the extracts:
Optimize your code: Try to optimize your code to use less memory. For example, if you're working with a large amount of data, consider processing it in batches rather than all at once. This was suggested by Varad Kabade in a forum discussion about a similar issue.
Use ObjectStore: If you're working with large datasets, consider storing some of your data in ObjectStore and retrieving it as needed. This can help to reduce the amount of memory your notebook uses at any one time. You can find more information about this in the same forum discussion.
Upgrade your research node: If optimizing your code and using ObjectStore doesn't solve the issue, you might need to upgrade to a research node with more memory. As Jared Broad mentioned in a forum discussion, the free tier of research nodes are capped at 2GB, which can be easily exceeded with substantial history requests.
Monitor your memory usage: Currently, it's not possible to track memory usage in the Research environment directly. However, Derek Melchin suggested in a forum discussion that you can track the amount of memory used when training your model locally to get an idea of how much memory you might need in the cloud.
I hope these suggestions help you to resolve the memory issues you're experiencing with your research node.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Zachary Wray
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!