Back

Downloading pickled models from dropbox in Python

Hi,

I have an sklearn model that I trained on my local machine and uploaded to dropbox as a pickle object. Is there anyway to use it in my backtests on QC?

I tried downloading using

response = urllib.urlopen("URL/to/model")
self.model = pickle.loads(response.read())

But got the following error:


Build Warning: File: n/a Line:0 Column:0 - Sorry, 'open(' is not whitelisted within QuantConnect.
Remove this code statement
Update Backtest








A robust way to work with such files for models is on the feature list(help welcome from us community members). I'm unsure if there is a big security issue with using pickled files from outside the cloud(such as locally trained Sklearn or Deap models), or even TensorFlow's binary checkpoint files. However, saving a model from a Research notebook, Backtest, or Live algo to ensure a persistable model will be great as the financial data exists in the QC cloud.

https://github.com/QuantConnect/Lean/issues/1040
0

I think something like this is allowed in C# (if I not totally reading this wrong):

https://www.quantconnect.com/forum/discussion/385/uploading-data-files

I was hoping there would be some equivalent code to this (as mentioned on the other discussion):

var client = new WebClient();
var json = client.DownloadString (address);
var customObject = JsonConvert.DeserializeObject(json);
0

Along those lines, you may have luck manually working with & iimporting your sklean model in json format:

https://cmry.github.io/notes/serialize

*I have not tested this, I just came across this url when trying to save to json vs pickle(only way suggested in sklearn docs).

0

Yeah, I could do that. But that still doesn't solve the problem of accessing a web url from within python.

0

Hi Ahmad; We've got a feature coming soon to replace fetcher. Its not merged yet but we'll deploy it this week.

2

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


The only way I know of accessing outside data via python is with the custom data importing from docs which is a bit heavy handed vs the quick json downlload mentioned in that other post. Perhaps others can suggest more ideas.

0

QCAlgorithm has the Download method that accepts an url and returns the data from that url. Here is an example.
The QuantConnect team hasn't tackled the issue #1040 that implemensts a virtual file system or object storage yet.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed