Dropbox download of keras TF H5 files in python?

Did you ever find a way to do this? I'd also like to import a trained H5 model into QC.

While not a solution for working with H5 files, I have found a way by to work with external keras models by just importing the saved model weights in json format from DropBox. You need to recreate the model and load the weights in QC, but it works as expected.

Hi Rob Anderson,
Thank you for your contribution. Would you be interested in sharing your implementation with the community?
Best,
Varad Kabade

Hi Rob Anderson,

I am also interested in your solution. Right now all my models have been trained and stored as h5 files locally. So I wonder if I can just load them again locally, then use the Model.to_json(**kwargs) method which is mentioned here to save the model weights as json and then upload and load the model weights as json file to QC with tf.keras.models.model_from_json(json_string, custom_objects=None) as described in the same link. 

As far as I understand, I need to copy paste the model architecture and compile the model with these pretrained weights in QC again to get the same model, right? Does that work with the Download function at the moment?

Maybe this here works. Will try it:

Ok, after some searching, I've read that the internal Keras to_json only exports the model but not the weights.

So for weights, I came across the following solution that copies the weights to numpy arrays or lists and then dumps them to files.

HI Michael and Varad,

I'll try to find the time to load a more complete answer, but in the meanwhile, here are the essential steps I followed. Hope this helps 😊

Step 1 - save model weights (Notebook)

I compiled and ran my model in Jupyter Notebook, e.g. essential code/steps…

import json
import keras

from keras.models import clone_model, Sequential
from keras.layers import Dense

def convert_weights_to_json(weights):
    weights = [w.tolist() for w in weights]
    json_weights = json.dumps(weights, separators=(',',':'))
    return json_weights
 
model = Sequential()
model.add(Dense(12, input_dim=3, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

model.compile(
    optimizer = 'adam', 
    loss      = 'binary_crossentropy', 
    metrics   = ['accuracy'],
    verbose   = 0
)

model.fit(x_train, y_train, epochs=150, validation_data = (x_val, y_val))

model.save('desired/path/model') # to save in standard TF format

weights = model.get_weights()

json_weights = convert_weights_to_json(weights)

with open('desired/path/weights.json', 'w') as file:
    file.write(json_weights) 

Step 2 - upload weights to Dropbox

Upload weights.json to Dropbox folder and copy Dropbox link.

Step s - load model weights (QC)

Here is an extract of workable code in my main.py file.

import json
import keras

class KerasModelLoad(QCAlgorithm):

    def Initialize(self):
        ...
    
    	# same architecture as model used to generate weights
		self.model = Sequential()
		self.model.add(Dense(12, input_dim=3, activation='relu'))
		self.model.add(Dense(8, activation='relu'))
		self.model.add(Dense(1, activation='sigmoid'))
		
		self.json_weights = self.Download('https://www.dropbox.com/s/[id]/weights.json?dl=1') 
		self.weights = self.json_to_weights(self.json_weights)
		
		self.model.set_weights(self.weights)
		
	def OnData(self, data):
		...
		
		x_input = ...
		prediction = self.model.predict(np.array([x_input,]), verbose=0)
		
		...
		
	def json_to_weights(self, json_weights):
        weights = json.loads(json_weights)
        weights = [np.asarray(x) for x in weights]
        return weights 

Notes:

• No need to compile model in QC, just need to make sure the model architecture is the same.
• If you need to use the same model, but load different weights (e.g. for different securities), you can define the model once and use clone_model to create and load individual weights.
• I have personally loaded multiple 20Mb models this way.

Sorry, the indentation got screwed up when saving my code examples… hopefully you can recreate successfully.

Thank you, Rob! Great work!

I managed to load the weights and get my LSTM predictions in Lean CLI local installation last night. Somehow I ran into issues with the Dropbox solution for online backtesting. It took the html code instead of the json. So your code is really appreciated and I'll check what mistake I did.

That's how I managed to save and load the weights locally:

1. In my LSTM code:
    

class NumpyArrayEncoder(JSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.ndarray):
            return obj.tolist()
        return JSONEncoder.default(self, obj)

and in the callbacks:

def on_epoch_end(self, epoch, logs=None):
        print('End epoch', epoch, 'of training')

        # Dump model weights
        weights_list = model.get_weights()
        print('write model weights into json file')
        with open('weights.json', "w") as write_file:
            json.dump(weights_list, write_file, cls=NumpyArrayEncoder)

2. In the Lean Python project, after creating the model structure:

def Initialize(self):
   # Load model structure from other def
   model = create_network()
   # Load weights from json
   path = os.path.join(Globals.DataFolder, 'weights.json')
   with open(path) as file:
       contents = file.read()
   decoded_weights = json.loads(contents)
   array_weights = np.array([np.array(x) for x in decoded_weights])
   model.set_weights(array_weights)

So now the LSTM is making predictions in the Lean CLI local backtest when it receives the LSTM windows. I just still need to save and load the scaler, probably in a similar way.

Hi Michael,

That's great news, glad to hear you're making progress!

The most likely cause of HTML being returned from Dropbox is not changing the last character in the URL to 1 to signal a download.

I like your implementation and will look at using your class for encoding (my previous attempts weren't all that successful). 

Regarding scaling, I have saved min/max values for series used during training to a separate json file and imported them in a similar way.

Cheers, Rob