Algorithm Reference

Universes

Introduction

Universe selection is the process of screening or filtering the assets you'd like to trade by some predetermined formula. This helps to avoid selection bias in your algorithm. The following section introduces how to use the QuantConnect Universe Selection API.

QuantConnect provides two universes; Coarse Universe, and Fine Universe for the US Equities Market. The QuantConnect Universe data is refreshed every day.

How Does Universe Selection Work

Universe Selection sends a large data set into a filter function. After passing through your filters the algorithm needs you to return an array of Symbol objects. LEAN automatically subscribes to these new symbols and adds them to your algorithm.

Your algorithm can do almost anything inside your filter functions, but the goal should be to narrow down the 8,000 daily stocks to a few which are most applicable for your algorithm.

The first stage of the filter is called "Coarse Universe Selection". This is for roughly filtering down the universe by relatively simple properties like price and volume. The result of this is piped into the Fine Universe Selection function to perform selection on fundamental data like PE Ratio, and Earnings.

By default assets selected by universe selection are requested with minute resolution data. You can change the default selection by adjusting the UniverseSettings property which we'll dive into below.

Coarse Universe Selection

Coarse Universe selection is the built in universe data provided by QuantConnect. Using financial data we generate a few key properties for each symbol and allow you to filter the universe of 16,400+ symbols to receive the symbols matching your filter criteria. It uses the CoarseFundamental type, which extends BaseData.

Coarse fundamental has the following properties you can use to perform rough filtering.

class CoarseFundamental : BaseData {
    public string Market;           // Market for symbol
    public long Volume;             // Traded shares
    public decimal DollarVolume;    // Traded shares x Price
    public decimal Price;           // Yesterday's close price
    public decimal PriceFactor;     // Price factor for the given date 
    public decimal SplitFactor;     // Split factor for the given date
    public decimal AdjustedPrice;   // Split and dividend adjusted price
    public Symbol Symbol;           // Asset symbol
    public bool HasFundamentalData; // Whether it has fundamental    
}
class CoarseFundamental(BaseData): 
    self.Volume               // Traded shares (double)
    self.DollarVolume         // Traded shares x Price (double)
    self.Price                // Yesterday's close price (double) 
    self.Symbol               // Asset symbol 
    self.HasFundamentalData   // Whether it has fundamental data (bool)

Coarse filtering allows you to select an unlimited universe of symbols to analyse. You are only limited by practical memory and speed limits and may quickly run out of memory if you backtest too many symbols in parallel. These limits can be increased with a subscription.

// Take the top 50 by dollar volume using coarse
AddUniverse(coarse => {
    return (from c in coarse
	    where c.Price > 10
	    orderby c.DollarVolume descending
            select c.Symbol).Take(50);
});
# In Initialize:
self.AddUniverse(self.CoarseSelectionFunction)

def CoarseSelectionFunction(self, coarse):
    '''Take the top 50 by dollar volume using coarse'''
    # sort descending by daily dollar volume
    sortedByDollarVolume = sorted(coarse, \
        key=lambda x: x.DollarVolume, reverse=True)

    # we need to return only the symbol objects
    return [ x.Symbol for x in sortedByDollarVolume[:50] ]
// We can use the following helper to reproduce the same result
AddUniverse(Universe.DollarVolume.Top(50));

Fundamentals Selection

QuantConnect provides MorningstarĀ® corporate fundamentals data for US Equities symbols. The available dataset includes about 5000 tickers with 900 fields starting from January 1998. It uses the FineFundamental type.

For the FineFundamental properties, please check out our data library page.

Like coarse universes, fine universes allow you to select an unlimited universe of symbols to analyse. You are only limited by practical memory and speed limits and may quickly run out of memory if you backtest too many symbols in parallel. These limits can be increased with a subscription. Also you must use coarse fundamental universe to narrow down the universe as a "pre-filter".

// Take the top 50 by dollar volume using coarse
// Then the top 10 by PERatio using fine
AddUniverse(
    coarse => {
        return (from c in coarse
            where c.Price > 10
            orderby c.DollarVolume descending
            select c.Symbol).Take(50);
    },
    fine => {
        return (from f in fine
            orderby f.ValuationRatios.PERatio descending
            select f.Symbol).Take(10);
    });
# In Initialize:
self.AddUniverse(self.CoarseSelectionFunction, self.FineSelectionFunction)

def CoarseSelectionFunction(self, coarse):
    '''Take the top 50 by dollar volume using coarse'''
    # sort descending by daily dollar volume
    sortedByDollarVolume = sorted(coarse, \
        key=lambda x: x.DollarVolume, reverse=True)

    # we need to return only the symbol objects
    return [ x.Symbol for x in sortedByDollarVolume[:50] ]

def FineSelectionFunction(self, fine):
    '''sort the data by P/E ratio and take the top 10 '''
    # sort descending by P/E ratio
    sortedByPeRatio = sorted(fine, \
        key=lambda x: x.ValuationRatios.PERatio, reverse=True)

    # take the top entries from our sorted collection
    return [ x.Symbol for x in sortedByPeRatio[:10] ]

Custom Universe Selection

Custom universes allow using an external data source as the security filtering source. Like normal custom data sources, custom universes are provided by extending BaseData. With this system you can define data formats to filter and select the data.

Each BaseData of the custom universe is 1 line of the source file. The Reader method will be called repeatedly until the date/time advances or the end of file is reached. This way you can group universe data together, and pass it as a single collection into the filter function.

AddUniverse("myCustomUniverse", Resolution.Daily, nyseTopGainersList => {
      return from singleStockData in nyseTopGainersList
             where singleStockData.Rank > 5
             select singleStockData.Symbol;
});
# In Initialize
self.AddUniverse(NyseTopGainers, "myCustomUniverse", Resolution.Daily, self.nyseTopGainers)

def nyseTopGainers(self, data):
    return [ x.Symbol for x in data if x["Rank"] > 5 ]
//Example custom universe data; it is virtually identical to other custom data types.
public class NyseTopGainers : BaseData {
    public int TopGainersRank;
    public override DateTime EndTime {
        // define end time as exactly 1 day after Time
	get { return Time + QuantConnect.Time.OneDay; }
	set { Time = value - QuantConnect.Time.OneDay; }
    }

    public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode) {
        return new SubscriptionDataSource(@"your-remote-universe-data", SubscriptionTransportMedium.RemoteFile);
     }

     public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode) {
         // Generate required data, then return an instance of your class.
        return new NyseTopGainers {
            Symbol = Symbol.Create(symbolString, SecurityType.Equity, Market.USA),
            Time = date,
            TopGainersRank = rank
        };
    }
}
# Example custom universe data; it is virtually identical to other custom data types.
class NyseTopGainers(PythonData):

    def GetSource(self, config, date, isLiveMode):
        return SubscriptionDataSource(@"your-remote-universe-data", SubscriptionTransportMedium.RemoteFile)

    def Reader(self, config, line, date, isLiveMode):
        # Generate required data, then return an instance of your class.
        nyse = NyseTopGainers()
        nyse.Time = date
        # define end time as exactly 1 day after Time
        nyse.EndTime = nyse.Time + timedelta(1)
        nyse.Symbol = Symbol.Create(symbolString, SecurityType.Equity, Market.USA)
        nyse["Rank"] = rank
        return nyse

Configuring Universe Securities

When adding securities from the universe, some algorithms need raw or partially adjusted price data. This can be controlled by the SetSecurityInitializer() method. With this method you can apply any fill model or special data requests on a per-security basis.

The most common request is for raw, unadjusted, price information. This can be achieved by combining the SetSecurityInitializer method with SetDataNormalizationMode() method. As each securities is added to the universe, its data will be set to any of the DataNormalizationMode enum values.

//In Initialize
SetSecurityInitializer(CustomSecurityInitializer);

private void CustomSecurityInitializer(Security security)
{
	//Initialize the security with raw prices
	security.SetDataNormalizationMode(DataNormalizationMode.Raw);
}
#In Initialize
self.SetSecurityInitializer(self.CustomSecurityInitializer)

def CustomSecurityInitializer(self, security):
    '''Initialize the security with raw prices'''
    security.SetDataNormalizationMode(DataNormalizationMode.Raw)

For simple requests you can use the functional implementation of the security initializer. This lets you configure and return the security object with 1 line of code:

//Most common request; requesting raw prices for universe securities.
SetSecurityInitializer(x => x.SetDataNormalizationMode(DataNormalizationMode.Raw));
# Most common request; requesting raw prices for universe securities.
self.SetSecurityInitializer(lambda x: x.SetDataNormalizationMode(DataNormalizationMode.Raw))

Option Universes

When you add an option to the algorithm it adds many many individual option contract securities. These are modelled as a "universe" of option contracts. We provide the SetFilter method to help narrow the option strike and expiry dates down to a range you are interested.

For more information on selecting options universes see the Options section in Data Library documentation.

Future Universes

In a similar way to options; when you add a futures asset to your algorithm it adds all the contracts which match your filter as a universe of futures contracts. The primary difference is that futures don't have a strike price so the universe filter is primarily focused on the future expiration date.

For more information on selecting futures universes see the Futures section in Data Library documentation.

You can also see our Tutorials and Videos. You can also get in touch with us via Chat.

Did you find this page Helpful ?