Machine Learning

Training Models

Introduction

Algorithms usually must process each timeslice within 10 minutes, but the Train method allows you to increase this time to train machine learning models. The length of time you can train depends on your training quotas.

Train Models

To train models immediately, call the Train method and pass in the name of your training method.

Train(MyTrainingMethod);
self.Train(self.MyMethod)

Immediate training is most useful for training your model when you first deploy your strategy to production or when the model's performance begins to degrade.

Schedule Training Sessions

You can schedule model training sessions in a similar way to a Scheduled Event. To schedule a training session do this, pass in a DateRules and TimeRules argument to the Train method.

// Set TrainingMethod to be executed at 8:00 am every Sunday
Train(DateRules.Every(DayOfWeek.Sunday), TimeRules.At(8, 0), MyTrainingMethod);
# Set TrainingMethod to be executed at 8:00 am every Sunday
self.Train(self.DateRules.Every(DayOfWeek.Sunday), self.TimeRules.At(8,0), self.MyTrainingMethod)

We recommend you schedule your training sessions for when the market is closed to get the best compute allocation. While the market is open, your CPU is occupied with processing incoming tick data and handling other LEAN events.

Training Quotas

Training resources are allocated with a leaky bucket algorithm where you can use a maximum of n-minutes in a single training session and the number of available minutes refills over time. This design gives you burst allocations when you need them and recharges the allowance to prepare for the next training.

Cloud Quotas

If you execute algorithms in QC Cloud, see Training Quotas for more information about the training quotas.

Local Quotas

If you execute algorithms locally, the following table shows the default settings for the leaky bucket algorithm:

SettingValue
Capacity (minutes)
120
Time interval (minutes)1440
Refill amount (minutes per time interval)
Capacity / 7

To allow virtually unlimited training for local algorithms, add the following key-value pairs to your Lean/Launcher/config.json file:

"scheduled-event-leaky-bucket-capacity" : 99999999,
"scheduled-event-leaky-bucket-time-interval-minutes" : 1,
"scheduled-event-leaky-bucket-refill-amount": 999999,

Check Model Readiness

In backtests, the Train method is synchronous, so it blocks your algorithm execution while the model trains. In live trading, the Train method is asynchronous, so ensure your model is trained before you continue the algorithm execution. Training occurs on a separate thread, so set a boolean flag to notify your algorithm of the model state. A semaphore is a thread-safe flag you can use to synchronize program operations across different threads.

class SemaphoreTrainingAlgorithm(QCAlgorithm):

    # Model Object
    model = None
    # Model State Flag
    model_is_training = False

    def Initialize(self) -> None: 
        self.Train(self.my_training_method)
    
    def my_training_method(self) -> None: 
        self.model_is_training = True
        # Perform Work
        self.model_is_training = False
    
    def OnData(self, slice: Slice) -> None: 
        # Do not use model while it is being trained.
        if self.model_is_training:
            return
        
        # Once training is complete; use the model safely.
        result = self.model.Predict()
public class SemaphoreTrainingAlgorithm : QCAlgorithm
{
    // Model Object
    private MachineLeaningModel _model;
    // Model State Flag
    private bool _modelIsTraining;

    public override void Initialize()
    {
        Train(MyTrainingMethod);
    }

    private void MyTrainingMethod()
    {
        _modelIsTraining = true;
        // Perform Work
        _modelIsTraining = false;
    }

    public override void OnData(Slice slice)
    {
        // Do not use model while it is being trained.
        if (_modelIsTraining)
        {
            return;
        }
        // Once training is complete; use the model safely.
        var result = _model.Predict();
    }
}

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: