Back

How to Load an External PyTorch Model?

First of all, when I import a library that I created (including the torch imports and architecture), I start getting the following error: "AttributeError: Oanda". I have seen that other people also experienced this error due to some imports:

https://www.quantconnect.com/forum/discussion/4742/addforex-market-oanda/p1/comment-13833

Apart from that, I would like to know how I can download and load an external PyTorch model. I have downloaded the file with Python requests but wasn't able to load the model with the same name that I have utilized for downloading. Then, I have tried self.Download, which is existing in some examples, but getting "File name too long" error while trying to load that into a PyTorch model. I guess that the variable to which I assigned self.Download output doesn't contain the filename. Hence, my question is how I can download an external file from Dropbox and then use that for loading to a PyTorch model as follows:

        file = self.Download("https://www.dropbox.com/.../best_model.pt?dl=0")
        model.load_state_dict(torch.load(file))
        model.eval()

Update Backtest







0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Note: I also basically need the same for loading a pretrained and pickled class from sklearn.preprocessing.

0

Hi Kamer. Attached you have a sample generic code about how you can read a pickle file stored on Dropbox. The pickle sample file I created just contains a dictionary, as it is an example. You could add more complex objects there.

Regarding PyTorch, I have no experience at all with that library. Nevertheless, I suggest you to train your model with exactly the same version that QuantConnect is using. You could check the version on DockerfileLeanFoundation.

https://github.com/QuantConnect/Lean/blob/master/DockerfileLeanFoundation

Exactly the same applies to your sklearn.preprocessing classes. Now, I am not sure if there's limit on the size of file you can get with requests. 

0


Thank you very much Gabriel, and have a great weekend!!! Your solution did not work neither unfortunately but at least gave me a proposal (through io.BytesIO) that I can investigate further.

AttributeError : 'bytes' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

0

Ok, I managed to solve the issue as follows: model.load_state_dict(torch.load(io.BytesIO(req.content)))

But it gave an UnpicklingError, probably due to older PyTorch version as you have guessed in advance!

0

Lean actually has PyTorch 1.0.0, that I am also using, please let me know in case of ideas on UnpicklingError.

0

I saved the model with Python 3.6.8 and PyTorch 1.0.1.post2 (very close versions to what exist in the Lean).

0

Resolved. I found the solution by developing an API for my PyTorch model and interacting with that, rather than trying to download and load the PyTorch model to the platform.

0

API version shared in the following issue, I can also share the Flask server code for the model if required by anybody:

https://www.quantconnect.com/forum/discussion/5428/converting-portfolio-balancing-classic-algorithm-to-a-framework-algorithm/p1

 

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed