Custom Universes

CSV Format Example

Introduction

This page explains how to import custom data for universe selection sourced in CSV format.

Data Format

You must create a file with data in CSV format. Ensure the data in the file is in chronological order.

20170704,SPY,QQQ,FB,AAPL,IWM
20170706,QQQ,AAPL,IWM,FB,GOOGL
20170707,IWM,AAPL,FB,BAC,GOOGL
...
20170729,SPY,QQQ,FB,AAPL,IWM
20170801,QQQ,FB,AAPL,IWM,GOOGL
20170802,QQQ,IWM,FB,BAC,GOOGL

Define Custom Types

To define a custom data type, inherit the BaseDataPythonData class and override the GetSource and Reader methods.

public class StockDataSource : BaseData
{
    public List<string> Symbols { get; set; } = new();
    public override DateTime EndTime => Time.AddDays(1);

    public override SubscriptionDataSource GetSource(
        SubscriptionDataConfig config,
        DateTime date,
        bool isLiveMode)
    {
        return new SubscriptionDataSource("https://www.dropbox.com/s/ae1couew5ir3z9y/daily-stock-picker-backtest.csv?dl=1", SubscriptionTransportMedium.RemoteFile);
    }

    public override BaseData Reader(
        SubscriptionDataConfig config,
        string line,
        DateTime date,
        bool isLiveMode)
    {
        var stocks = new StockDataSource();

        try
        {
            var csv = line.Split(',');
            stocks.Time = DateTime.ParseExact(csv[0], "yyyyMMdd", null);
            stocks.Symbols.AddRange(csv.Skip(1));
        }
        catch { return null; }

        return stocks;
    }
}
class StockDataSource(PythonData):

    def GetSource(self,
         config: SubscriptionDataConfig,
         date: datetime,
         isLive: bool) -> SubscriptionDataSource:
        return SubscriptionDataSource("https://www.dropbox.com/s/ae1couew5ir3z9y/daily-stock-picker-backtest.csv?dl=1", SubscriptionTransportMedium.RemoteFile)

    def Reader(self,
         config: SubscriptionDataConfig,
         line: str,
         date: datetime,
         isLive: bool) -> BaseData:

        if not (line.strip() and line[0].isdigit()): return None
        
        stocks = StockDataSource()
        stocks.Symbol = config.Symbol

        try:
            csv = line.split(',')
            stocks.Time = datetime.strptime(csv[0], "%Y%m%d")
            stocks.EndTime = stocks.Time + timedelta(days=1)
            stocks["Symbols"] = csv[1:]

        except ValueError:
            # Do nothing
            return None

        return stocks

Initialize Universe

To perform a universe selection with custom data, in the Initialize method, call the AddUniverse method.

public class MyAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        AddUniverse<StockDataSource>("myStockDataSource", Resolution.Daily, FilterFunction);
    }
}
class MyAlgorithm(QCAlgorithm): 
    def Initialize(self) -> None:
        self.AddUniverse(StockDataSource, "my-stock-data-source", Resolution.Daily, self.FilterFunction)
    

Receive Custom Data

As your data reader reads your custom data file, LEAN adds the data points into a List[StockDataSource])IEnumerable<StockDataSource> object it passes to your algorithm's filter function. Your filter function needs to return a list of Symbol or strstring object. LEAN automatically subscribes to these new assets and adds them to your algorithm.

public class MyAlgorithm : QCAlgorithm
{
    private IEnumerable<string> FilterFunction(IEnumerable<StockDataSource> stockDataSource)
    {
        return stockDataSource.SelectMany(x => x.Symbols);
    }
}
class MyAlgorithm(QCAlgorithm):
    def FilterFunction(self, data: List[StockDataSource]) -> List[str]:
        list = []
        for item in data:
            for symbol in item["Symbols"]:
                list.append(symbol)
        return list
    

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: