trophy

Long Only, ETF Alpha Streams Competition with $27,500 Prize Pool

Back

Uploading data files?

Hey all,

so one of my models requires a large-ish calibration (e.g. the parameters for my statistical model) - this is on the order of a few megabytes.
Is there any easy way to upload data files into a project & then load then from the running code?

Cheers,
Thomas
Update Backtest








0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


If it suits you could put it into a dropbox / ftp and allow the algorithm to access it? Then downloading from there with a WebClient request -

Or, if its stored as a time-value set of data you can use the custom data importers & event handlers to get the configuration delivered to an event handler over time - this is an example importing custom data.
1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey there,

ok, the external storage sounds like a plan for now :-). This *may* become more cumbersome as the models grow larger, but let's cross that bridge once we reach it :-)

Thanks again for the quick and great support ! :)

Cheers,
Thomas
0

Hey there,

If there's already a plan to add a feature for persistent data files, I'd be a big fan - the remote
download option isn't great (authentication / crypto etc. ends up being complicated), and my attempt
at simply using large literal strings seems to be thwarted by Mono not liking literal strings over a certain
size.

Cheers,
Thomas
0

Thanks Thomas - can you import it with the QC event system? And read it in one line at a time as a time-value series? I like persistent uploads or cloud-internal stored uploads for faster loading, but they wouldn't be as dynamic & harder to update than a FTP/Web sourced file.

How big is the file out of interest? Our event loader system allows breaking up files into periods and referencing different files depending on the date of the backtest.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey Jared,

I'll look into the event system - but using it feels like abusing it ;) as this is literally just a few megabytes of text with parameters for a model.

Regarding FTP/Web sourcing: I tried this route, but it is made more complicated by the fact that the entire System.IO package is missing, so classes like StreamReader etc. are unavailable.

Is it possible to whitelist individual classes in the System.IO package perhaps?

Cheers,
Thomas
0

Regarding your question on file size: For the initial test, the file is roughly 4 megabytes, but once more than 50 stock streams become available the model spec is a few hundred megs of text (and roughly 200m of RAM once loaded & made useful).

Cheers,
Thomas
0

Ok perfect, definitely need to load it in via web client in this case. Could you encode the data in JSON and decode it to a Model/Array? Just add using Newtonsoft.Json; and make a class which models your data format -

var client = new WebClient();
var json = client.DownloadString (address);
var customObject = JsonConvert.DeserializeObject(json);
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Cool, WebClient works great for me :)
0

After running into the same issue - trying to read data from a file, but System.IO is disallowed - I found this article.

I'm getting an error trying to use WebClient. Has it been decommissioned in the last 5 years?

The error is: "The type or namespace name 'WebClient' could not be found (are you missing a using directive or an assembly reference?)"

I have included "using Newtonsoft.Json;" at the top of my C# file. 

0

Hi Chad Crenshaw ,

Like System.IO, System.Net.WebClient is also disallowed for security reasons.
Please use Download method that downloads a URL and returns it as a string.

// If using dropbox remember to add the &dl=1 to trigger a download
var file = Download("https://www.dropbox.com?....&dl=1");

 

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed