Bit of a newbie so apologies if I'm asking a silly question. Live deployment of a crypto machine learning algo, throwing up an error when I try and trigger the training function of the algo


Runtime Error: Algorithm took longer than 10 minutes on a single time loop. CurrentTimeStepElapsed: 0.0 minutes Stack Trace: System.TimeoutException: Algorithm took longer than 10 minutes on a single time loop. CurrentTimeStepElapsed: 0.0 minutes at QuantConnect.Isolator.MonitorTask (System.Threading.Tasks.Task task, System.TimeSpan timeSpan, System.Func`1[TResult] withinCustomLimits, System.Int64 memoryCap, System.Int32 sleepIntervalMillis) [0x002d3] in :0 at QuantConnect.Isolator.ExecuteWithTimeLimit (System.TimeSpan timeSpan, System.Func`1[TResult] withinCustomLimits, System.Action codeBlock, System.Int64 memoryCap, System.Int32 sleepIntervalMillis, QuantConnect.Util.WorkerThread workerThread) [0x00092] in :0 at QuantConnect.Lean.Engine.Engine.Run (QuantConnect.Packets.AlgorithmNodePacket job, QuantConnect.Lean.Engine.AlgorithmManager manager, System.String assemblyPath, QuantConnect.Util.WorkerThread workerThread) [0x009f0] in :0

User: 108607, Project: 5001223, Algorithm: L-886773484792a4b50dd08fc926320cd6

The code in question triggers at 3AM

def NeuralNetworkTraining(self):
'''Train the Neural Network and save the model in the ObjectStore'''
symbols = list(self.modelBySymbol.keys())

if len(symbols) == 0:
self.Debug("no contracts found")

for symbol in symbols:
# Hourly historical data is used to train the machine learning model
history = self.History(symbol, (self.lookback + self.timesteps), Resolution.Hour)
self.Debug("Failed to receive history")
#history = self.x_scaler.fit_transform(history)

if 'open' in history and 'close' in history and 'high' in history and 'low' in history:
history = np.column_stack((history['open'], history['close'], history['high'], history['low']))
#history = np.column_stack((history['open']))

if len(history) < self.lookback:
self.Debug("Error while collecting the training data")

#history = list([i[0] for i in history])

self.Debug("Start Training for symbol {0}".format(symbol))

#First convert the data into 3D Array with (x train samples, 60 timesteps, 1 feature)
x_train = []
y_train = []
for i in range(self.timesteps, len(history)):
x_train.append(history[i - self.timesteps:i])

x_train, y_train = np.array(x_train), np.array(y_train)
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 4))
y_train = np.reshape(y_train, (y_train.shape[0], 1))
if np.any(np.isnan(x_train)):
self.Debug("Error in Training Data")
if np.any(np.isnan(y_train)):
self.Debug("Error in Validation Data")

x_scaler = MinMaxScaler(feature_range=(0, 1))
y_scaler = MinMaxScaler(feature_range=(0, 1))

x_train = x_scaler.fit_transform(x_train.reshape(-1, x_train.shape[-1])).reshape(x_train.shape)
#x_train = self.x_scaler.fit_transform(x_train)
y_train = y_scaler.fit_transform(y_train)


# build a neural network from the 1st layer to the last layer
model = Sequential()

model.add(Dense(10, input_dim = 1))

sgd = SGD(lr = 0.01) # learning rate = 0.01

# choose loss function and optimizing method
model.compile(loss='mse', optimizer=sgd)
if symbol in self.modelBySymbol and self.modelBySymbol[symbol] is not None:
model = self.modelBySymbol[symbol]
iterations = 1
#If Model not exist for symbol then create one
opt_cells = 5
model = Sequential()

model.add(LSTM(units = opt_cells, return_sequences = True, input_shape = (x_train.shape[1], 4)))


model.add(LSTM(units = opt_cells, return_sequences = True))

model.add(LSTM(units = opt_cells, return_sequences = True))

model.add(LSTM(units = opt_cells, return_sequences = False))

model.add(Dense(1, activation='linear'))

adam = Adam(lr=0.001, clipnorm=1.0)
model.compile(loss='mean_squared_error', optimizer=adam, metrics=['accuracy'])

iterations = 50
# pick an iteration number large enough for convergence
for step in range(iterations):
# training the model
#cost = model.train_on_batch(predictor, predictand)
hist =, y_train, epochs = 1) #verbose=0,
acc = list(hist.history['accuracy'])[-1]
loss = list(hist.history['loss'])[-1]

self.scalersBySymbol[symbol] = (x_scaler, y_scaler)
self.modelBySymbol[symbol] = model
self.Debug("End Training for symbol {0} with accuracy {1}".format(symbol, acc))

According to the log, my algo pulls the data and then starts the training. The runtime error is then generated. 

Is this simply a function of the fact I'm on the $20/month plan with limited ML time?  Do I need to move to the algo with LEAN deployed on a beefier server?

Appreciated your time.