Algorithm Reference

Universes

Introduction

Often you may want to rotate securities in your algorithm based on filter criteria. You may only want equities above their 200 day EMA, or to only follow stocks on your custom list of symbols. This is possible using our universe selection API.

Universes are how LEAN organizes collections of data subscriptions under the hood. Each security and its data is controlled by a universe. When no universes are asking for data, the asset is removed from your algorithm. If your algorithm has open orders or holdings in a security we will not remove it from your subscriptions.

Every algorithm has a hidden "user-defined-universe". The assets in this universe are set by the AddEquity / AddForex methods. These assets are fixed and never removed from your algorithm.

Universes (and data QuantConnect provides) are refreshed every day by default. Custom universes can be refreshed as often as required. This is controlled in the algorithm.UniverseSettings variable which we'll describe in more detail below.

Basic Universe API

Adding Universes
Universes are added using the AddUniverse() method API. They are a type of data subscription which control what subscriptions are requested; and as such you can create custom universe data types. Depending on what type of universe you are adding there are many helper methods to make it easier. AddUniverse() methods take function filters as parameters, these filters must return an enumerable of Symbol objects.

Universe Settings
If you do not pass in a full universe object the UniverseSettings property is used to fill in the blanks. Changing the UniverseSettings algorithm property can be helpful to simplify adding universes. Universes have 5 key properties:

    //Popular universe settings:
    UniverseSettings.Resolution      // What resolution should added assets use
                    .Leverage        // What leverage should assets use in the universe?
                    .Fillforward     // Should asset data fill forward?
                    .MinimumTimeInUniverse // Minimum time assets should be in universe
                    .ExtendedMarketHours  // Should assets also feed extended market hours?
    //Popular universe settings:
    self.UniverseSettings.Resolution      // What resolution should added assets use
                    .Leverage        // What leverage should assets use in the universe?
                    .Fillforward     // Should asset data fill forward?
                    .MinimumTimeInUniverse // Minimum time assets should be in universe
                    .ExtendedMarketHours  // Should assets also feed extended market hours?

Once added universes are stored in the algorithm.UniverseManager.

Universe Events
When universe contents are changed (securities are added or removed from the algorithm) we generate an OnSecuritiesChanged event. This allows your algorithm to know the changes in the universe state. The event passes in the SecurityChanges object containing references to the Added and Removed securities.

// Inside your initialize function:

// Subscriptions added via universe selection will have this resolution
UniverseSettings.Resolution = Resolution.Hour;

// Force securities to remain in the universe for a minimum of 30 days
UniverseSettings.MinimumTimeInUniverse = TimeSpan.FromDays(30);

// Helper: Add US-equity universe for the top 50 stocks by dollar volume
AddUniverse(Universe.DollarVolume.Top(50));

// Helper: Add US-equity universe for the bottom 50 stocks by dollar volume
AddUniverse(Universe.DollarVolume.Bottom(50));

// Helper: Add US-equity universe for the 90th dollar volume percentile
AddUniverse(Universe.DollarVolume.Percentile(90));

// Helper: Add US-equity universe for stocks between the 70th and 80th dollar volume percentile
AddUniverse(Universe.DollarVolume.Percentile(70, 80));
// Overriding the on changes event handler.
public override void OnSecuritiesChanged(SecurityChanges changes)
{
     _changes = changes;
}
# Overriding the on changes event handler.
def OnSecuritiesChanged(self, changes):
     self._changes = changes
// Use UniverseManager to find symbols from the Universe Selection
foreach (var universe in UniverseManager.Values)
{
    // User defined universe has symbols from AddSecurity/AddEquity calls
    if (universe is UserDefinedUniverse)
    {
        continue;
    }
    var symbols = universe.Members.Keys;
}
# Use UniverseManager to find symbols from the Universe Selection
for universe in self.UniverseManager.Values:

    # User defined universe has symbols from AddSecurity/AddEquity calls
    if universe is UserDefinedUniverse:
        continue

    symbols = universe.Members.Keys

Coarse Universe Selection

Coarse Universe selection is the built in universe data provided by QuantConnect. Using financial data we generate a few key properties for each symbol and allow you to filter the universe of 16,400+ symbols to receive the symbols matching your filter criteria. It uses the CoarseFundamental type, which extends BaseData.

Coarse fundamental has the following properties you can use to perform rough filtering.

class CoarseFundamental : BaseData {
    public string Market;           // Market for symbol
    public long Volume;             // Traded shares
    public decimal DollarVolume;    // Traded shares x Price
    public decimal Price;           // Yesterday's close price
    public decimal PriceFactor;     // Price factor for the given date 
    public decimal SplitFactor;     // Split factor for the given date
    public decimal AdjustedPrice;   // Split and dividend adjusted price
    public Symbol Symbol;           // Asset symbol
    public bool HasFundamentalData; // Whether it has fundamental    
}

Coarse filtering allows you to select an unlimited universe of symbols to analyse. You are only limited by practical memory and speed limits and may quickly run out of memory if you backtest too many symbols in parallel. These limits can be increased with a subscription.

// Take the top 50 by dollar volume using coarse
AddUniverse(coarse => {
    return (from c in coarse
	    where c.Price > 10
	    orderby c.DollarVolume descending
            select c.Symbol).Take(50);
});
# In Initialize:
self.AddUniverse(self.CoarseSelectionFunction)

def CoarseSelectionFunction(self, coarse):
    '''Take the top 50 by dollar volume using coarse'''
    # sort descending by daily dollar volume
    sortedByDollarVolume = sorted(coarse, \
        key=lambda x: x.DollarVolume, reverse=True)

    # we need to return only the symbol objects
    return [ x.Symbol for x in sortedByDollarVolume[:50] ]
// We can use the following helper to reproduce the same result
AddUniverse(Universe.DollarVolume.Top(50));

Fundamentals Selection

QuantConnect provides MorningstarĀ® corporate fundamentals data for US Equities symbols. The available dataset includes about 5000 tickers with 900 fields starting from January 1998. It uses the FineFundamental type.

For the FineFundamental properties, please check out our data library page.

Like coarse universes, fine universes allow you to select an unlimited universe of symbols to analyse. You are only limited by practical memory and speed limits and may quickly run out of memory if you backtest too many symbols in parallel. These limits can be increased with a subscription. Also you must use coarse fundamental universe to narrow down the universe as a "pre-filter".

// Take the top 50 by dollar volume using coarse
// Then the top 10 by PERatio using fine
AddUniverse(
    coarse => {
        return (from c in coarse
            where c.Price > 10
            orderby c.DollarVolume descending
            select c.Symbol).Take(50);
    },
    fine => {
        return (from f in fine
            orderby f.ValuationRatios.PERatio descending
            select f.Symbol).Take(10);
    });
# In Initialize:
self.AddUniverse(self.CoarseSelectionFunction, self.FineSelectionFunction)

def CoarseSelectionFunction(self, coarse):
    '''Take the top 50 by dollar volume using coarse'''
    # sort descending by daily dollar volume
    sortedByDollarVolume = sorted(coarse, \
        key=lambda x: x.DollarVolume, reverse=True)

    # we need to return only the symbol objects
    return [ x.Symbol for x in sortedByDollarVolume[:50] ]

def FineSelectionFunction(self, fine):
    '''sort the data by P/E ratio and take the top 10 '''
    # sort descending by P/E ratio
    sortedByPeRatio = sorted(fine, \
        key=lambda x: x.ValuationRatios.PERatio, reverse=True)

    # take the top entries from our sorted collection
    return [ x.Symbol for x in sortedByPeRatio[:10] ]

Custom Universe Selection

Custom universes allow using an external data source as the security filtering source. Like normal custom data sources, custom universes are provided by extending BaseData. With this system you can define data formats to filter and select the data.

Each BaseData of the custom universe is 1 line of the source file. The Reader method will be called repeatedly until the date/time advances or the end of file is reached. This way you the engine can group universe data together, and pass it as a single collection into the filter function.

AddUniverse("myCustomUniverse", Resolution.Daily, nyseTopGainersList => {
      return from singleStockData in nyseTopGainersList
             where singleStockData.Rank > 5
             select singleStockData.Symbol;
});
# In Initialize
self.AddUniverse(NyseTopGainers, "myCustomUniverse", Resolution.Daily, self.nyseTopGainers)

def nyseTopGainers(self, data):
    return [ x.Symbol for x in data if x["Rank"] > 5 ]
//Example custom universe data; it is virtually identical to other custom data types.
public class NyseTopGainers : BaseData {
    public int TopGainersRank;
    public override DateTime EndTime {
        // define end time as exactly 1 day after Time
	get { return Time + QuantConnect.Time.OneDay; }
	set { Time = value - QuantConnect.Time.OneDay; }
    }

    public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode) {
        return new SubscriptionDataSource(@"your-remote-universe-data", SubscriptionTransportMedium.RemoteFile);
     }

     public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode) {
         // Generate required data, then return an instance of your class.
        return new NyseTopGainers {
            Symbol = Symbol.Create(symbolString, SecurityType.Equity, Market.USA),
            Time = date,
            TopGainersRank = rank
        };
    }
}
# Example custom universe data; it is virtually identical to other custom data types.
class NyseTopGainers(PythonData):

    def GetSource(self, config, date, isLiveMode):
        return SubscriptionDataSource(@"your-remote-universe-data", SubscriptionTransportMedium.RemoteFile)

    def Reader(self, config, line, date, isLiveMode):
        # Generate required data, then return an instance of your class.
        nyse = NyseTopGainers()
        nyse.Time = date
        # define end time as exactly 1 day after Time
        nyse.EndTime = nyse.Time + timedelta(1)
        nyse.Symbol = Symbol.Create(symbolString, SecurityType.Equity, Market.USA)
        nyse["Rank"] = rank
        return nyse

Configuring Universe Securities

When adding securities from the universe, some algorithms need raw or partially adjusted price data. This can be controlled by the SetSecurityInitializer() method. With this method you can apply any fill model or special data requests on a per-security basis.

The most common request is for raw, unadjusted, price information. This can be achieved by combining the SetSecurityInitializer method with SetDataNormalizationMode() method. As each securities is added to the universe, its data will be set to any of the DataNormalizationMode enum values.

//In Initialize
SetSecurityInitializer(CustomSecurityInitializer);

private void CustomSecurityInitializer(Security security)
{
	//Initialize the security with raw prices
	security.SetDataNormalizationMode(DataNormalizationMode.Raw);
}
#In Initialize
self.SetSecurityInitializer(self.CustomSecurityInitializer)

def CustomSecurityInitializer(self, security):
    '''Initialize the security with raw prices'''
    security.SetDataNormalizationMode(DataNormalizationMode.Raw)

For simple requests you can use the functional implementation of the security initializer. This lets you configure and return the security object with 1 line of code:

//Most common request; requesting raw prices for universe securities.
SetSecurityInitializer(x => x.SetDataNormalizationMode(DataNormalizationMode.Raw));
# Most common request; requesting raw prices for universe securities.
self.SetSecurityInitializer(lambda x: x.SetDataNormalizationMode(DataNormalizationMode.Raw))

Option Universes

When you add an option to the algorithm it adds many many individual option contract securities. These are modelled as a "universe" of option contracts. We provide the SetFilter method to help narrow the option strike and expiry dates down to a range you are interested.

For more information on selecting options universes see the Options section in Data Library documentation.

Future Universes

In a similar way to options; when you add a futures asset to your algorithm it adds all the contracts which match your filter as a universe of futures contracts. The primary difference is that futures don't have a strike price so the universe filter is primarily focused on the future expiration date.

For more information on selecting futures universes see the Futures section in Data Library documentation.

You can also see our Tutorials and Videos. You can also get in touch with us via Chat.

Did you find this page Helpful ?