Back

Persistent Storage and Scalable Compute Power?

I appologize if these questions have been answered...I'm new here and did my best with the search.

I'm interested in training machine learning models and I run into two potential issues:

1) Lack of persistent storage. Is there some and I'm just missing it? I would really like a way to save trained network parameters. Retraining the model every time you spin it up is really not practical (especially in light of #2, below). 

I realize that storage external to QC is not an option because of the risk of users attempting to exfiltrate proprietary data. However, I'm willing to pay for QC internal storage at typical cloud rates (+reasonable markup). I am intersted in equities, so I need to train models on QC to get access to the data. I realize that this creates "lock-in" but given the access to these data sets, it seems like a fair trade.

2) Lack of CPU scaling or GPUs. It's really nice (and often necessary) to accelerate training with more power. Again, I'm willing to pay for the resources should they be made available.

Final comment of my first post: I've poured through all of the API documentation and I must say that's IMHO it's really well designed. It's very easy to understand and use. And extra kudos for opensourcing LEAN! I'm really excited to work within the QC infrastructure, using LEAN. Thank you, QuantConnect.

Update Backtest







0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Strange nobody answered your question. I am looking for answers for the same thing. Did you happen to find a way to have persistent storage?

1

Hi Sameer Abdul and GregasMaximus ,

First of all, sorry about missing this thread,

We are about to implement persistent storage. Please check out the following GitHub pull-request:
Add IObjectStore/LocalObjectStore/QCAlgorithm.ObjectStore #3911

QuantConnect provides additional RAM if necessary. Please take a look at the options on the pricing page.

On top of that, we have implemented helpers for machine learning training (that takes over 10 minutes which is Lean timeout), There is a new section in the docs about it: Machine Learning.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed