Share New Research 
Hello, I want to download raw bytes of the url because I am having issues when trying to load the data into a Pytorch model.
data_str = self.Download("https://www.dropbox.com/s/examplepath/checkpoint.pth?dl=1")
data_byte = data_str.encode()
self.agent.local.load_state_dict(torch.load(BytesIO(data_byte), map_location=lambda storage, loc: storage))
From what I understand, the self.Download method saves the data as a decoded string? So I have to conver the string back to byte data by encoding the data with 'UTF-8' codec. When I load data_byte, it returns UnpicklingError: invalid load key, '<' using the method above. I was able to load my models previously using request and just getting the raw content iteself with no encoding.
url = "https://www.dropbox.com/s/examplepth/checkpoint.pth?dl=1"
data = requests.get(url)
self.agent.local.load_state_dict(torch.load(BytesIO(data.content), map_location=lambda storage, loc: storage))Any idea on how to convert the self.Downloaded string back to byte data?
Jared Broad
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Joe Bastulli
Thanks for the fast response Jared! However im still a little confused about how I should encode the downloaded file. Does this look right? It is still giving me a UnpicklingError: invalid load key, '<' Error. Thanks for all of the help so far!
data_str = self.Download("https://www.dropbox.com/s/examplepath/checkpoint.pth?dl=1") # String Encode to bytes data_b = data_str.encode("UTF-8") # Base64 Encode the bytes data_e = base64.b64encode(data_b) # Decoding the Base64 bytes to string data_s1 = data_e.decode("UTF-8") # Encoding the Base64 encoded string into bytes data_b1 = data_s1.encode("UTF-8") # Decoding the Base64 bytes data_d = base64.b64decode(data_b1) self.agent.local.load_state_dict(torch.load(BytesIO(data_d), map_location=lambda storage, loc: storage))Joe Bastulli
Ahh, I got it. We have to convert the file to Base64 and save that file to dropbox, then download and reconvert!
Joe Bastulli
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!