Back

Self.Download encoding issues.

Hello, I want to download raw bytes of the url because I am having issues when trying to load the data into a Pytorch model.

data_str = self.Download("https://www.dropbox.com/s/examplepath/checkpoint.pth?dl=1")

data_byte = data_str.encode()

self.agent.local.load_state_dict(torch.load(BytesIO(data_byte), map_location=lambda storage, loc: storage))

 

From what I understand, the self.Download method saves the data as a decoded string?  So I have to conver the string back to byte data by encoding the data with 'UTF-8' codec. When I load data_byte, it returns UnpicklingError: invalid load key, '<' using the method above.  I was able to load my models previously using request and just getting the raw content iteself with no encoding.

 

url = "https://www.dropbox.com/s/examplepth/checkpoint.pth?dl=1"

data = requests.get(url)

self.agent.local.load_state_dict(torch.load(BytesIO(data.content), map_location=lambda storage, loc: storage))

Any idea on how to convert the self.Downloaded string back to byte data?

Update Backtest







0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


You'll have to convert it to base64 and back to bytes
1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks for the fast response Jared!  However im still a little confused about how I should encode the downloaded file.  Does this look right?  It is still giving me a UnpicklingError: invalid load key, '<' Error.  Thanks for all of the help so far!

 

data_str = self.Download("https://www.dropbox.com/s/examplepath/checkpoint.pth?dl=1")

# String Encode to bytes
data_b = data_str.encode("UTF-8")

# Base64 Encode the bytes
data_e = base64.b64encode(data_b)

# Decoding the Base64 bytes to string
data_s1 = data_e.decode("UTF-8")

# Encoding the Base64 encoded string into bytes
data_b1 = data_s1.encode("UTF-8")

# Decoding the Base64 bytes
data_d = base64.b64decode(data_b1)


self.agent.local.load_state_dict(torch.load(BytesIO(data_d), map_location=lambda storage, loc: storage))

 

0

Ahh, I got it.  We have to convert the file to Base64 and save that file to dropbox, then download and reconvert!

1

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed