Universes

Chained Universes

Introduction

You can combine ("chain") universes together to fetch fundamental and alternative data on a specific subset of assets. Universes filter input data and return Symbol objects. The only requirement is that Symbol objects the filter returns are a subset of the input data. The source of the Symbol objects is unrestricted, so you can feed the output of one universe into another.

Filter Pattern

Universes filter a large set of Symbol objects by a coarse filter to quickly reduce the data processing requirement. This is often a first step before applying a second filter or requesting alternative data. For example, a strategy might only be interested in easily tradable liquid assets so quickly eliminates all stocks with less than $1M USD / day in trading volume.

A chain of universe blocks that get smaller as they are filtered over time

The order of your filters can improve the speed of your research. By applying filters that narrow the universe the most, or are the lightest weight first, you can significantly reduce the amount of data your algorithm processes. Unless necessary, you can also not return any selections from earlier filters to further improve research speed, keeping only the universe data for later filters.

Universe Data Weights

To speed up your algorithm, request the lightest weight data first before chaining heavier filters or adding alternative data. The following table shows the size each dataset:

NameData Size / Weight
US Equities (Coarse)Light (1 GB)
US Equities (Fine/Fundamental)Heavy (5 TB)
US Equity Options Huge (200 TB)
US Index Options Medium (500 GB)
US Futures Medium (500 GB)
US Futures OptionsMedium (500 GB)
CryptoLight (1 GB)
Alternative / GeneralLight (100 MB - 2 GB)
Alternative / Tiingo News Medium (200 GB)

Universe Schedules

Universes typically run overnight and are available before market open. Universes are not currently "ordered", so universe chaining works best with slower universes. For example, use a slow-changing ETF Constituents Universe to set the Symbol list for alternative data.

Chaining Coarse and Alternative Data

The following example chains the QuiverQuantTwitterFollowersUniverse alternative universe to the coarse universe. It first selects the 100 most liquid US Equities and then filters them down based on their Twitter followers number and weekly change. The output of the alternative universe selection method is the output of the chained universe. 

using System.Collections.Generic;
using System.Linq;
using QuantConnect.Data;
using QuantConnect.Data.UniverseSelection;
using QuantConnect.DataSource;
using QuantConnect.Securities;

namespace QuantConnect.Algorithm.CSharp
{
    public class ChainedUniverseAlgorithm : QCAlgorithm
    {
        private List<Symbol> _coarse = new();
        private Universe _universeCoarse;
        private Universe _universeTwitter;

        public override void Initialize()
        {
            SetStartDate(2023, 1, 2);
            SetCash(100000);

            _universeCoarse = AddUniverse(CoarseFilterFunction);
            _universeTwitter = AddUniverse<QuiverQuantTwitterFollowersUniverse>(
                "QuiverQuantTwitterFollowersUniverse", Resolution.Daily, FollowerSelection);
        }

        private IEnumerable<Symbol> CoarseFilterFunction(IEnumerable<CoarseFundamental> coarse)
        {
            _coarse = (from c in coarse
                       orderby c.DollarVolume descending
                       select c.Symbol).Take(100).ToList();
            return Universe.Unchanged;
        }

        private IEnumerable<Symbol> FollowerSelection(IEnumerable<QuiverQuantTwitterFollowersUniverse> altCoarse)
        {
            var followers = from d in altCoarse
                         where d.Followers > 200000m && d.WeekPercentChange > 0m
                         select d.Symbol;
            return _coarse.Intersect(followers);
        }

        public override void OnSecuritiesChanged(SecurityChanges changes)
        {
            foreach (var added in changes.AddedSecurities)
            {
                AddData<QuiverQuantTwitterFollowers>(added.Symbol);
            }
        }

        public override void OnData(Slice data)
        {
            foreach (var kvp in data.Get<QuiverQuantTwitterFollowers>())
            {
                var datasetSymbol = kvp.Key;
                var dataPoint = kvp.Value;
                Debug($"{datasetSymbol} followers at {data.Time}: {dataPoint.Followers}");
            }
        }
    }
}
from AlgorithmImports import * 

class ChainedUniverseAlgorithm(QCAlgorithm):

    coarse = []
    universe_coarse = None
    universe_twitter = None

    def Initialize(self):
        self.SetCash(100000)
        self.SetStartDate(2023, 1, 2)
        self.universe_coarse = self.AddUniverse(self.CoarseFilterFunction)
        self.universe_twitter = self.AddUniverse(QuiverQuantTwitterFollowersUniverse, "QuiverQuantTwitterFollowersUniverse", Resolution.Daily, self.FollowerSelection)

    def CoarseFilterFunction(self, coarse: List[CoarseFundamental]) -> List[Symbol]:
        sorted_by_dollar_volume = sorted(coarse, key=lambda x: x.DollarVolume, reverse=True) 
        self.coarse = [c.Symbol for c in sorted_by_dollar_volume[:100]]
        return Universe.Unchanged

    def FollowerSelection(self, alt_coarse: List[QuiverQuantTwitterFollowersUniverse]) -> List[Symbol]:
        self.followers = [d.Symbol for d in alt_coarse if d.Followers > 200000 and d.WeekPercentChange > 0]
        return list(set(self.coarse) & set(self.followers))

    def OnSecuritiesChanged(self, changes):
        for added in changes.AddedSecurities:
            self.AddData(QuiverQuantTwitterFollowers, added.Symbol)

    def OnData(self, data):
        # Prices in the slice from the universe selection
        # Alternative data in slice from OnSecuritiesChanged Addition
        # for ticker,bar in data.Bars.items():
        #     pass
        for dataset_symbol, data_point in data.Get(QuiverQuantTwitterFollowers).items():
            self.Debug(f"{dataset_symbol} followers at {data.Time}: {data_point.Followers}")

Chaining ETF Universe and Alt Data

The following example chains a QuiverQuantTwitterFollowersUniverse alternative universe to the SPY ETF universe. It first selects all constituents of SPY and then filters them down with based on their Twitter followers number and weekly change. The output of the alternative universe selection method is the output of the chained universe. 

using System.Collections.Generic;
using System.Linq;
using QuantConnect.Data;
using QuantConnect.Data.UniverseSelection;
using QuantConnect.DataSource;
using QuantConnect.Securities;

namespace QuantConnect.Algorithm.CSharp
{
    public class ChainedUniverseAlgorithm : QCAlgorithm
    {
        private List<Symbol> _etf = new();
        private Universe _universeEtf;
        private Universe _universeTwitter;

        public override void Initialize()
        {
            SetStartDate(2023, 1, 2);
            SetCash(100000);

            _universeEtf = AddUniverse(Universe.ETF("SPY", Market.USA, UniverseSettings, ETFConstituentsFilter));
            // or var symbol = QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA);
            // _universeEtf = AddUniverseSelection(new ETFConstituentsUniverseSelectionModel(
            //    symbol, UniverseSettings, ETFConstituentsFilter));
            _universeTwitter = AddUniverse<QuiverQuantTwitterFollowersUniverse>(
                "QuiverQuantTwitterFollowersUniverse", Resolution.Daily, FollowerSelection);
        }

        private IEnumerable<Symbol> ETFConstituentsFilter(IEnumerable<ETFConstituentData> constituents)
        {
            _etf = constituents.Select(c => c.Symbol).ToList();
            return Universe.Unchanged;
        }

        private IEnumerable<Symbol> FollowerSelection(IEnumerable<QuiverQuantTwitterFollowersUniverse> altCoarse)
        {
            var followers = from d in altCoarse
                            where d.Followers > 200000m && d.WeekPercentChange > 0m
                            select d.Symbol;
            return _etf.Intersect(followers);
        }

        public override void OnSecuritiesChanged(SecurityChanges changes)
        {
            foreach (var added in changes.AddedSecurities)
            {
                AddData<QuiverQuantTwitterFollowers>(added.Symbol);
            }
        }

        public override void OnData(Slice data)
        {
            foreach (var kvp in data.Get<QuiverQuantTwitterFollowers>())
            {
                var datasetSymbol = kvp.Key;
                var dataPoint = kvp.Value;
                Debug($"{datasetSymbol} followers at {data.Time}: {dataPoint.Followers}");
            }
        }
    }
}
from AlgorithmImports import * 

class ChainedUniverseAlgorithm(QCAlgorithm):

    etf = []
    universe_etf = None
    universe_twitter = None

    def Initialize(self):
        self.SetCash(100000)
        self.SetStartDate(2023, 1, 2)
        
        self.universe_etf = self.AddUniverse(self.Universe.ETF("SPY", Market.USA, self.UniverseSettings, self.ETFConstituentsFilter))
        # or symbol = Symbol.Create("SPY", SecurityType.Equity, Market.USA)
        # self.universe_etf = self.AddUniverseSelection(ETFConstituentsUniverseSelectionModel(
        #   symbol, self.UniverseSettings, self.ETFConstituentsFilter))
        self.universe_twitter = self.AddUniverse(QuiverQuantTwitterFollowersUniverse, 
            "QuiverQuantTwitterFollowersUniverse", Resolution.Daily, self.FollowerSelection)

    def ETFConstituentsFilter(self, constituents: List[ETFConstituentData]) -> List[Symbol]:
        self.etf = [c.Symbol for c in constituents]
        return Universe.Unchanged

    def FollowerSelection(self, alt_coarse: List[QuiverQuantTwitterFollowersUniverse]) -> List[Symbol]:
        self.followers = [d.Symbol for d in alt_coarse if d.Followers > 200000 and d.WeekPercentChange > 0]
        return list(set(self.etf) & set(self.followers))

    def OnSecuritiesChanged(self, changes):
        for added in changes.AddedSecurities:
            self.AddData(QuiverQuantTwitterFollowers, added.Symbol)

    def OnData(self, data):
        # Prices in the slice from the universe selection
        # Alternative data in slice from OnSecuritiesChanged Addition
        # for ticker,bar in data.Bars.items():
        #     pass
        for dataset_symbol, data_point in data.Get(QuiverQuantTwitterFollowers).items():
            self.Debug(f"{dataset_symbol} followers at {data.Time}: {data_point.Followers}")

Chaining to US Equity Options

The following example chains an OptionFilterUniverse to the QQQ ETF universe. It first selects the 30 largest-weighted constituents of QQQ and then selects their call Option contracts that expire within 60 days.

using System.Collections.Generic;
using System.Linq;
using QuantConnect.Data;
using QuantConnect.Data.UniverseSelection;

namespace QuantConnect.Algorithm.CSharp
{
    public class ChainedUniverseAlgorithm : QCAlgorithm
    {
        private Universe _universeEtf;
        private List<Symbol> _optionContracts = new();

        public override void Initialize()
        {
            SetStartDate(2023, 2, 2);
            SetCash(100000);

            UniverseSettings.DataNormalizationMode = DataNormalizationMode.Raw;
            _universeEtf = AddUniverse(Universe.ETF("QQQ", Market.USA, UniverseSettings, ETFConstituentsFilter));
        }

        private IEnumerable<Symbol> ETFConstituentsFilter(IEnumerable<ETFConstituentData> constituents)
        {
            return (from c in constituents
                    orderby c.Weight descending
                    select c.Symbol).Take(10);
        }

        public override void OnSecuritiesChanged(SecurityChanges changes)
        {
            foreach (var added in changes.AddedSecurities)
            {
                if (added.Type != SecurityType.Equity) continue;

                var contracts = OptionChainProvider.GetOptionContractList(added.Symbol, Time);
                var filteredContracts = (from contract in contracts
                                        where contract.ID.OptionRight == OptionRight.Call
                                        && contract.ID.StrikePrice >= 100
                                        && contract.ID.Date <= Time.AddDays(10)
                                        && contract.ID.OptionStyle == OptionStyle.American
                                        select AddOptionContract(contract).Symbol);
                _optionContracts = _optionContracts.Union(filteredContracts).ToList();
            }
            
            foreach (var removed in changes.RemovedSecurities)
            {
                if (_optionContracts.Contains(removed.Symbol))
                {
                    _optionContracts.Remove(removed.Symbol);
                }
            }
        }

        public override void OnData(Slice data)
        {
            foreach (var chain in data.OptionChains)
            {
                var symbol = chain.Key;
                foreach (var contract in chain.Value)
                {
                    Debug($"Found {contract.Symbol} option contract for {symbol}");
                }
            }
        }
    }
}
from AlgorithmImports import *

class ChainedUniverseAlgorithm(QCAlgorithm):

    universe_etf = None
    option_contracts = []

    def Initialize(self):
        self.SetCash(100000)
        self.SetStartDate(2023, 2, 2)
        
        self.UniverseSettings.DataNormalizationMode = DataNormalizationMode.Raw
        self.universe_etf = self.AddUniverse(self.Universe.ETF("QQQ", Market.USA, self.UniverseSettings, self.ETFConstituentsFilter))

    def ETFConstituentsFilter(self, constituents: List[ETFConstituentData]) -> List[Symbol]:
        sorted_by_weight = sorted(constituents, key=lambda x: x.Weight, reverse=True) 
        return [c.Symbol for c in sorted_by_weight[:10]]

    def OnSecuritiesChanged(self, changes: SecurityChanges) -> None:
        for added in changes.AddedSecurities:
            if added.Type != SecurityType.Equity: continue

            contracts = self.OptionChainProvider.GetOptionContractList(added.Symbol, self.Time)
            filtered_contracts = [self.AddOptionContract(contract).Symbol for contract in contracts \
                                    if contract.ID.OptionRight == OptionRight.Call \
                                    and contract.ID.StrikePrice >= 100 \
                                    and contract.ID.Date <= self.Time + timedelta(10) \
                                    and contract.ID.OptionStyle == OptionStyle.American]
            self.option_contracts = self.option_contracts + filtered_contracts

        for removed in changes.RemovedSecurities:
            if removed.Symbol in self.option_contracts:
                self.option_contracts.remove(removed.Symbol)

    def OnData(self, data: Slice) -> None:
        for chain in data.OptionChains:
            symbol = chain.Key
            for contract in chain.Value:
                self.Debug(f"Found {contract.Symbol} option contract for {symbol}")

Chaining ETF and Fundamental Universe

The following example chains an ETF constituents universe with a fundamental universe to select ETF constituents with a low PE ratio. 

using System.Collections.Generic;
using System.Linq;
using QuantConnect.Data;
using QuantConnect.Data.Fundamental;
using QuantConnect.Data.UniverseSelection;
using QuantConnect.Securities;

namespace QuantConnect.Algorithm.CSharp
{
    public class ChainedUniverseAlgorithm : QCAlgorithm
    {
        private Universe _universeEtf;
        private Universe _universeFine;

        public override void Initialize()
        {
            SetStartDate(2023, 1, 2);
            SetCash(100000);

            _universeEtf = Universe.ETF("QQQ", Market.USA, UniverseSettings, ETFConstituentsFilter);
            _universeFine = AddUniverse(_universeEtf, FineSelection);
        }

        private IEnumerable<Symbol> ETFConstituentsFilter(IEnumerable<ETFConstituentData> constituents)
        {
            return constituents.Select(c => c.Symbol);
        }

        private IEnumerable<Symbol> FineSelection(IEnumerable<FineFundamental> fine)
        {
            return (from f in fine
                    orderby f.ValuationRatios.PERatio
                    select f.Symbol).Take(20);
        }

        public override void OnData(Slice data)
        {
            foreach (var symbol in data.Keys)
            {
                Debug($"{symbol} PE Ratio: {Securities[symbol].Fundamentals.ValuationRatios.PERatio}");
            }
        }
    }
}
from AlgorithmImports import * 

class ChainedUniverseAlgorithm(QCAlgorithm):

    universe_etf = None
    universe_fine = None

    def Initialize(self):
        self.SetCash(100000)
        self.SetStartDate(2023, 2, 2)
        
        self.universe_etf = self.Universe.ETF("QQQ", Market.USA, self.UniverseSettings, self.ETFConstituentsFilter)
        self.universe_fine = self.AddUniverse(self.universe_etf, self.FineSelection)

    def ETFConstituentsFilter(self, constituents: List[ETFConstituentData]) -> List[Symbol]:
        return [c.Symbol for c in constituents]

    def FineSelection(self, fine: List[FineFundamental]) -> List[Symbol]:
        sorted_by_pe_ratio = sorted(fine, key=lambda f: f.ValuationRatios.PERatio)
        return [f.Symbol for f in sorted_by_pe_ratio[:20]]

    def OnData(self, data):
        for symbol in data.Keys:
            self.Debug(f"{symbol} PE Ratio: {self.Securities[symbol].Fundamentals.ValuationRatios.PERatio}")

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: