Bit of a newbie so apologies if I'm asking a silly question. Live deployment of a crypto machine learning algo, throwing up an error when I try and trigger the training function of the algo
Runtime Error: Algorithm took longer than 10 minutes on a single time loop. CurrentTimeStepElapsed: 0.0 minutes Stack Trace: System.TimeoutException: Algorithm took longer than 10 minutes on a single time loop. CurrentTimeStepElapsed: 0.0 minutes at QuantConnect.Isolator.MonitorTask (System.Threading.Tasks.Task task, System.TimeSpan timeSpan, System.Func`1[TResult] withinCustomLimits, System.Int64 memoryCap, System.Int32 sleepIntervalMillis) [0x002d3] in :0 at QuantConnect.Isolator.ExecuteWithTimeLimit (System.TimeSpan timeSpan, System.Func`1[TResult] withinCustomLimits, System.Action codeBlock, System.Int64 memoryCap, System.Int32 sleepIntervalMillis, QuantConnect.Util.WorkerThread workerThread) [0x00092] in :0 at QuantConnect.Lean.Engine.Engine.Run (QuantConnect.Packets.AlgorithmNodePacket job, QuantConnect.Lean.Engine.AlgorithmManager manager, System.String assemblyPath, QuantConnect.Util.WorkerThread workerThread) [0x009f0] in :0
User: 108607, Project: 5001223, Algorithm: L-886773484792a4b50dd08fc926320cd6
The code in question triggers at 3AM
def NeuralNetworkTraining(self):
'''Train the Neural Network and save the model in the ObjectStore'''
symbols = list(self.modelBySymbol.keys())
if len(symbols) == 0:
self.Debug("no contracts found")
return
for symbol in symbols:
try:
# Hourly historical data is used to train the machine learning model
history = self.History(symbol, (self.lookback + self.timesteps), Resolution.Hour)
self.Debug(history)
except:
self.Debug("Failed to receive history")
#history = self.x_scaler.fit_transform(history)
if 'open' in history and 'close' in history and 'high' in history and 'low' in history:
history = np.column_stack((history['open'], history['close'], history['high'], history['low']))
#history = np.column_stack((history['open']))
if len(history) < self.lookback:
self.Debug("Error while collecting the training data")
continue
#history = list([i[0] for i in history])
self.Debug("Start Training for symbol {0}".format(symbol))
#First convert the data into 3D Array with (x train samples, 60 timesteps, 1 feature)
x_train = []
y_train = []
for i in range(self.timesteps, len(history)):
x_train.append(history[i - self.timesteps:i])
y_train.append([history[i][0]])
x_train, y_train = np.array(x_train), np.array(y_train)
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 4))
y_train = np.reshape(y_train, (y_train.shape[0], 1))
if np.any(np.isnan(x_train)):
self.Debug("Error in Training Data")
continue
if np.any(np.isnan(y_train)):
self.Debug("Error in Validation Data")
continue
x_scaler = MinMaxScaler(feature_range=(0, 1))
y_scaler = MinMaxScaler(feature_range=(0, 1))
x_train = x_scaler.fit_transform(x_train.reshape(-1, x_train.shape[-1])).reshape(x_train.shape)
#x_train = self.x_scaler.fit_transform(x_train)
y_train = y_scaler.fit_transform(y_train)
#self.Debug(x_train.shape)
#self.Debug(y_train.shape)
#self.Debug(y_train)
# build a neural network from the 1st layer to the last layer
'''
model = Sequential()
model.add(Dense(10, input_dim = 1))
model.add(Activation('relu'))
model.add(Dense(1))
sgd = SGD(lr = 0.01) # learning rate = 0.01
# choose loss function and optimizing method
model.compile(loss='mse', optimizer=sgd)
'''
if symbol in self.modelBySymbol and self.modelBySymbol[symbol] is not None:
model = self.modelBySymbol[symbol]
iterations = 1
else:
#If Model not exist for symbol then create one
opt_cells = 5
model = Sequential()
model.add(LSTM(units = opt_cells, return_sequences = True, input_shape = (x_train.shape[1], 4)))
model.add(Dropout(0.2))
model.add(LSTM(units = opt_cells, return_sequences = True))
model.add(Dropout(0.2))
model.add(LSTM(units = opt_cells, return_sequences = True))
model.add(Dropout(0.2))
model.add(LSTM(units = opt_cells, return_sequences = False))
model.add(Dropout(0.2))
model.add(Dense(1, activation='linear'))
adam = Adam(lr=0.001, clipnorm=1.0)
model.compile(loss='mean_squared_error', optimizer=adam, metrics=['accuracy'])
iterations = 50
# pick an iteration number large enough for convergence
for step in range(iterations):
# training the model
#cost = model.train_on_batch(predictor, predictand)
hist = model.fit(x_train, y_train, epochs = 1) #verbose=0,
acc = list(hist.history['accuracy'])[-1]
loss = list(hist.history['loss'])[-1]
self.scalersBySymbol[symbol] = (x_scaler, y_scaler)
self.modelBySymbol[symbol] = model
self.Debug("End Training for symbol {0} with accuracy {1}".format(symbol, acc))
According to the log, my algo pulls the data and then starts the training. The runtime error is then generated.
Is this simply a function of the fact I'm on the $20/month plan with limited ML time? Do I need to move to the algo with LEAN deployed on a beefier server?
Appreciated your time.