Back

Uploading data files?

Hey all,

so one of my models requires a large-ish calibration (e.g. the parameters for my statistical model) - this is on the order of a few megabytes.
Is there any easy way to upload data files into a project & then load then from the running code?

Cheers,
Thomas
Update Backtest








If it suits you could put it into a dropbox / ftp and allow the algorithm to access it? Then downloading from there with a WebClient request -

Or, if its stored as a time-value set of data you can use the custom data importers & event handlers to get the configuration delivered to an event handler over time - this is an example importing custom data.
1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey there,

ok, the external storage sounds like a plan for now :-). This *may* become more cumbersome as the models grow larger, but let's cross that bridge once we reach it :-)

Thanks again for the quick and great support ! :)

Cheers,
Thomas
0

Hey there,

If there's already a plan to add a feature for persistent data files, I'd be a big fan - the remote
download option isn't great (authentication / crypto etc. ends up being complicated), and my attempt
at simply using large literal strings seems to be thwarted by Mono not liking literal strings over a certain
size.

Cheers,
Thomas
0

Thanks Thomas - can you import it with the QC event system? And read it in one line at a time as a time-value series? I like persistent uploads or cloud-internal stored uploads for faster loading, but they wouldn't be as dynamic & harder to update than a FTP/Web sourced file.

How big is the file out of interest? Our event loader system allows breaking up files into periods and referencing different files depending on the date of the backtest.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey Jared,

I'll look into the event system - but using it feels like abusing it ;) as this is literally just a few megabytes of text with parameters for a model.

Regarding FTP/Web sourcing: I tried this route, but it is made more complicated by the fact that the entire System.IO package is missing, so classes like StreamReader etc. are unavailable.

Is it possible to whitelist individual classes in the System.IO package perhaps?

Cheers,
Thomas
0

Regarding your question on file size: For the initial test, the file is roughly 4 megabytes, but once more than 50 stock streams become available the model spec is a few hundred megs of text (and roughly 200m of RAM once loaded & made useful).

Cheers,
Thomas
0

Ok perfect, definitely need to load it in via web client in this case. Could you encode the data in JSON and decode it to a Model/Array? Just add using Newtonsoft.Json; and make a class which models your data format -

var client = new WebClient();
var json = client.DownloadString (address);
var customObject = JsonConvert.DeserializeObject(json);
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Cool, WebClient works great for me :)
0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed