Custom Universes

JSON Format Example

Introduction

This page explains how to import custom data for universe selection sourced in JSON format.

Data Format

You must create a file with data in JSON format. Ensure the data in the file is in chronological order.

[
    {
      "Date": "20170704",
      "Symbols": ["SPY", "QQQ", "FB", "AAPL", "IWM"]
    },
    {
      "Date": "20170706",
      "Symbols": ["QQQ", "AAPL", "IWM", "FB", "GOOGL"]
    },
    ...
    {
      "Date": "20170801",
      "Symbols": ["QQQ", "FB", "AAPL", "IWM", "GOOGL"]
    },
    {
      "Date": "20170802",
      "Symbols": ["QQQ", "IWM", "FB", "BAC", "GOOGL"]
    }
]

Define Custom Types

To define a custom data type, inherit the BaseDataPythonData class and override the GetSource and Reader methods.

If you need to create multiple objects in your Readerreader method from a single line, follow these steps:

  1. In the GetSourceget_source method, pass FileFormat.UnfoldingCollectionFileFormat.UNFOLDING_COLLECTION as the third argument to the SubscriptionDataSource constructor.
  2. In the Readerreader method, order the objects by their timestamp and then return a BaseDataCollection(endTime, config.Symbol, objects)BaseDataCollection(end_time, config.symbol, objects) where objects is a list of your custom data objects.
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

public class StockDataSource : BaseData
{
    public List<Symbol> Symbols { get; set; } = [];
    public override DateTime EndTime => Time.AddDays(1);
    public string Line { get; set; }

    public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
    {
        if (!isLiveMode)
        {
             return new SubscriptionDataSource("universe-example.json", SubscriptionTransportMedium.ObjectStore, FileFormat.UnfoldingCollection);
        }
        return new SubscriptionDataSource("https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/universe-example.json", SubscriptionTransportMedium.RemoteFile, FileFormat.UnfoldingCollection);
    }

    public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
    {
        List<StockDataSource> objects = [];
        foreach(var obj in JsonConvert.DeserializeObject(line) as JArray)
        {
            var stocks = new StockDataSource()
            {
                Time = isLiveMode
                    ? DateTime.UtcNow.ConvertFromUtc(config.DataTimeZone) 
                    : DateTime.ParseExact(obj["Date"].Value<string>(), "yyyyMMdd", null),
                Line = JsonConvert.SerializeObject(obj)
            };

            stocks.Symbols.AddRange(obj["Symbols"].Values<string>().Select(ticker =>
            {
                // The tickers are point-in-time. We generate its security identifier for a given date
                // Then we create a Symbol where the Value is the ticker
                var sid = SecurityIdentifier.GenerateEquity(ticker, Market.USA, mappingResolveDate: stocks.Time);
                return new Symbol(sid, ticker);
            }));

            objects.Add(stocks);
        }

        return new BaseDataCollection(objects.Last().EndTime, config.Symbol, objects);
    }
}
class StockDataSource(PythonData):
    def get_source(self, config: SubscriptionDataConfig, date: datetime, is_live: bool) -> SubscriptionDataSource:
        if not is_live:
            return SubscriptionDataSource("universe-example.json", SubscriptionTransportMedium.OBJECT_STORE, FileFormat.UNFOLDING_COLLECTION)
        return SubscriptionDataSource("https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/universe-example.json", SubscriptionTransportMedium.REMOTE_FILE, FileFormat.UNFOLDING_COLLECTION)

    def reader(self, config: SubscriptionDataConfig, line: str, date: datetime, is_live: bool) -> BaseData:
        def convert(obj):
            stocks = StockDataSource()
            stocks.time = datetime.strptime(obj["Date"], "%Y%m%d")
            def point_in_time(ticker):
                # The tickers are point-in-time. We generate its security identifier for a given date
                # Then we create a Symbol where the value is the ticker
                sid = SecurityIdentifier.generate_equity(ticker, Market.USA, mapping_resolve_date=stocks.time)
                return Symbol(sid, ticker)
            stocks.symbols = [point_in_time (ticker) for ticker in obj["Symbols"]]
            return stocks

        objects = [convert(obj) for obj in json.loads(line)]

        return BaseDataCollection(objects[-1].end_time, config.symbol, objects)

Initialize Universe

To perform a universe selection with custom data, in the Initializeinitialize method, call the AddUniverseadd_universe method.

public class MyAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        AddUniverse<StockDataSource>(FilterFunction);
    }
}
class MyAlgorithm(QCAlgorithm): 
    def initialize(self) -> None:
        self.add_universe(StockDataSource, self._filter_function)

Receive Custom Data

As your data reader reads your custom data file, LEAN adds the data points into a List[StockDataSource])IEnumerable<StockDataSource> object it passes to your algorithm's filter function. Your filter function needs to return a list of Symbol or strstring object. LEAN automatically subscribes to these new assets and adds them to your algorithm.

public class MyAlgorithm : QCAlgorithm
{
    private IEnumerable<Symbol> FilterFunction(IEnumerable<BaseData> data)
    {
        var stockDataSource = data.OfType<StockDataSource>();
        return stockDataSource.SelectMany(x => x.Symbols);
    }

    public override void OnSecuritiesChanged(SecurityChanges changes)
    {
        Debug(changes.ToString());
    }
}
class MyAlgorithm(QCAlgorithm):
    def _filter_function(self, data: list[BaseData]) -> list[str]:
        symbols = []
        for item in data:
            symbols.extend(item.symbols)
        return symbols
    
    def on_securities_changed(self, changes: SecurityChanges) -> None:
        self.debug(str(changes))

If you add custom properties to your data object in the Readerreader method, LEAN adds them as members to the data object in your filter method. To ensure the property names you add in the Readerreader method follow the convention of member names, LEAN applies the following changes to the property names you provide in the Readerreader method:

  1. - and . characters are replaced with whitespace.
  2. The first letter is capitalized.
  3. Whitespace characters are removed.

For example, if you set a property name in the Readerreader method to ['some-property.name'], you can access it in your filter method through the Somepropertyname member of your data object.

Demonstration Algorithms

DropboxBaseDataUniverseSelectionAlgorithm.py Python DropboxBaseDataUniverseSelectionAlgorithm.cs C#

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: