Custom Universes

JSON Format Example

Introduction

This page explains how to import custom data for universe selection sourced in JSON format.

Data Format

You must create a file with data in JSON format. Ensure the data in the file is in chronological order.

[
    {
      "Date": "20170704",
      "Symbols": ["SPY", "QQQ", "FB", "AAPL", "IWM"]
    },
    {
      "Date": "20170706",
      "Symbols": ["QQQ", "AAPL", "IWM", "FB", "GOOGL"]
    },
    ...
    {
      "Date": "20170801",
      "Symbols": ["QQQ", "FB", "AAPL", "IWM", "GOOGL"]
    },
    {
      "Date": "20170802",
      "Symbols": ["QQQ", "IWM", "FB", "BAC", "GOOGL"]
    }
]

Define Custom Types

To define a custom data type, inherit the BaseDataPythonData class and override the GetSource and Reader methods.

If you need to create multiple objects in your Reader method from a single line, follow these steps:

  1. In the GetSource method, pass FileFormat.UnfoldingCollection as the third argument to the SubscriptionDataSource constructor.
  2. In the Reader method, order the objects by their timestamp and then return a BaseDataCollection(endTime, config.Symbol, objects) where objects is a list of your custom data objects.
using Newtonsoft.Json;

public class StockDataSource : BaseData
{
    [JsonProperty("Symbols")]
    public List<string> Symbols { get; set; }
    [JsonProperty("Date")]
    public string Date;
    public override DateTime EndTime => Time.AddDays(1);

    public override SubscriptionDataSource GetSource(
        SubscriptionDataConfig config,
        DateTime date,
        bool isLiveMode)
    {
        return new SubscriptionDataSource("https://www.dropbox.com/s/7xe7lfac52mdfpe/custom-universe.json?dl=1", 
                                          SubscriptionTransportMedium.RemoteFile,
                                          FileFormat.UnfoldingCollection);
    }

    public override BaseData Reader(
        SubscriptionDataConfig config,
        string line,
        DateTime date,
        bool isLiveMode)
    {
        var objects = JsonConvert.DeserializeObject<List<StockDataSource>>(line).Select(stocks =>
        {
            stocks.Time = DateTime.ParseExact(stocks.Date, "yyyyMMdd", null);
            return stocks;
        }).ToList();

        return new BaseDataCollection(objects.Last().EndTime, config.Symbol, objects);
    }
}
class StockDataSource(PythonData):

    def GetSource(self,
         config: SubscriptionDataConfig,
         date: datetime,
         isLive: bool) -> SubscriptionDataSource:
        return SubscriptionDataSource("https://www.dropbox.com/s/7xe7lfac52mdfpe/custom-universe.json?dl=1", 
                                      SubscriptionTransportMedium.RemoteFile,
                                      FileFormat.UnfoldingCollection)

    def Reader(self,
         config: SubscriptionDataConfig,
         line: str,
         date: datetime,
         isLive: bool) -> BaseData:
        
        objects = []
        data = json.loads(line)

        for datum in data:
            stocks = StockDataSource()
            stocks.Symbol = config.Symbol

            stocks.Time = datetime.strptime(datum["Date"], "%Y%m%d")
            stocks.EndTime = stocks.Time + timedelta(1)
            stocks["Symbols"] = datum["Symbols"]
            objects.append(stocks)

        return BaseDataCollection(objects[-1].EndTime, config.Symbol, objects)

Initialize Universe

To perform a universe selection with custom data, in the Initialize method, call the AddUniverse method.

public class MyAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        AddUniverse<StockDataSource>("myStockDataSource", Resolution.Daily, FilterFunction);
    }
}
class MyAlgorithm(QCAlgorithm): 
    def Initialize(self) -> None:
        self.AddUniverse(StockDataSource, "my-stock-data-source", Resolution.Daily, self.FilterFunction)
    

Receive Custom Data

As your data reader reads your custom data file, LEAN adds the data points into a List[StockDataSource])IEnumerable<StockDataSource> object it passes to your algorithm's filter function. Your filter function needs to return a list of Symbol or strstring object. LEAN automatically subscribes to these new assets and adds them to your algorithm.

public class MyAlgorithm : QCAlgorithm
{
    private IEnumerable<string> FilterFunction(IEnumerable<StockDataSource> stockDataSource)
    {
        return stockDataSource.SelectMany(x => x.Symbols);
    }
}
class MyAlgorithm(QCAlgorithm):
    def FilterFunction(self, data: List[StockDataSource]) -> List[str]:
        list = []
        for item in data:
            for symbol in item["Symbols"]:
                list.append(symbol)
        return list
    

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: