Datasets

Custom Data

Introduction

Running the LEAN engine locally with the CLI requires you to have your own local data. Besides market data, LEAN also supports importing custom data into your algorithm. This page explains how to access data from local files in your algorithm when running the LEAN engine locally.

Import Local Files

When running LEAN locally using the CLI, you can already use all of the features explained in the Importing Data page of the LEAN documentation. However, sometimes you may not want to upload your file to a cloud storage service like Dropbox. You can follow these steps to convert your custom data class (the class extending BaseDataPythonData) to retrieve data from a local file instead of a remote one:

  1. Copy the data file that you want to use to the data directory in your organization workspace.
  2. Open the source file containing your custom data class in an editor.
  3. Update your GetSourceget_source method to load data from the local file in your data directory. Make sure you only use forward slashes. Backward slashes as path separators don't work. For the Weather example in the LEAN documentation, that is done like this:
    using System;
    using System.IO;
    using QuantConnect.Data;
    
    namespace QuantConnect.Algorithm.CSharp
    {
        public class Weather : BaseData
        {
            public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLive)
            {
                // Old:
                // var source = "https://www.dropbox.com/s/8v6z949n25hyk9o/custom_weather_data.csv?dl=1";
                // return new SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile);
    
                // New:
                // Replace custom_weather_data.csv with the path to your data file in the data directory
                var source = Path.Combine(Globals.DataFolder, "custom_weather_data.csv");
                return new SubscriptionDataSource(source, SubscriptionTransportMedium.LocalFile);
            }
        }
    }
    import os
    
    from QuantConnect import Globals, SubscriptionTransportMedium
    from QuantConnect.Data import SubscriptionDataSource
    from QuantConnect.Python import PythonData
    
    
    class Weather(PythonData):
        def get_source(self, config: SubscriptionDataConfig, date: datetime, is_live: bool) -> SubscriptionDataSource:
            # Old:
            # source = "https://www.dropbox.com/s/8v6z949n25hyk9o/custom_weather_data.csv?dl=1"
            # return SubscriptionDataSource(source, SubscriptionTransportMedium.REMOTE_FILE)
    
            # New:
            # Replace custom_weather_data.csv with the path to your data file in the data directory
            source = os.path.join(Globals.DATA_FOLDER, "custom_weather_data.csv")
            return SubscriptionDataSource(source, SubscriptionTransportMedium.LOCAL_FILE)
  4. Save the source file.

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: