QuantConnect

Factors

Introduction

Factors by QuantConnect provides precomputed factors for individual US Equities plus unlinked factors that aren't tied to a tradable security, such as US Treasury yield curve rates. The data starts in 1998, covers every US Equity that has appeared in the SPY ETF constituents universe since August 2015, and is delivered on a daily frequency. This dataset is created by computing each factor from our own datasets with the same LEAN tooling an algorithm would use, so backtest and precomputed values match exactly.

For more information about the Factors dataset, including CLI commands and pricing, see the dataset listing.

About the Provider

QuantConnect was founded in 2012 to serve quants everywhere with the best possible algorithmic trading technology. Seeking to disrupt a notoriously closed-source industry, QuantConnect takes a radically open-source approach to algorithmic trading. Through the QuantConnect web platform, more than 50,000 quants are served every month.

Getting Started

The following snippet demonstrates how to request data from the Factors dataset:

# Factors linked to a specific stock:
self._equity = self.add_equity("AAPL", Resolution.DAILY)
self._equity.factors = self.add_data(Factors, self._equity.symbol).symbol
# Unlinked factors:
self._unlinked_factors = self.add_data(Factors, Factors.UNLINKED).symbol
# All factors:
self._universe = self.add_universe(FactorsUniverse, self._select_assets)
// Factors linked to a specific stock:
_equity = AddEquity("SPY", Resolution.Daily);
_equityFactors = AddData<Factors>(_equity.Symbol).Symbol;
// Unlinked factors:
_unlinkedFactors = AddData<Factors>(Factors.Unlinked).Symbol;
// All factors:
_universe = AddUniverse<FactorsUniverse>(SelectAssets);

Data Summary

The following table describes the dataset properties:

PropertyValue
Start DateJanuary 1998
Data DensityDense
Asset CoverageAll US Equities in the SPY ETF, plus unlinked factors
ResolutionDaily
TimezoneNew York

Requesting Data

To add Factors data to your algorithm, call the AddDataadd_data method. For the factors linked to a specific Equity, pass the underlying Equity Symbol. For the unlinked factors, which aren't tied to any tradable security, pass Factors.UnlinkedFactors.UNLINKED. Save a reference to each dataset Symbol so you can access the data later in your algorithm.

class FactorsDataAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2014, 3, 25)
        self.set_end_date(2014, 4, 8)
        self.set_cash(100000)
        self._equity = self.add_equity("SPY", Resolution.DAILY)
        self._equity.factors = self.add_data(Factors, self._equity.symbol).symbol
        self._unlinked_factors = self.add_data(Factors, Factors.UNLINKED).symbol
public class FactorsDataAlgorithm : QCAlgorithm
{
    private Equity _equity;
    private Symbol _equityFactors, _unlinkedFactors;

    public override void Initialize()
    {
        SetStartDate(2014, 3, 25);
        SetEndDate(2014, 4, 8);
        SetCash(100000);
        _equity = AddEquity("SPY", Resolution.Daily);
        _equityFactors = AddData<Factors>(_equity.Symbol).Symbol;
        _unlinkedFactors = AddData<Factors>(Factors.Unlinked).Symbol;
    }
}

Accessing Data

To get the current factor data, index the current Slice with the dataset Symbol. Slice objects deliver unique events to your algorithm as they happen, but the Slice may not contain data for your dataset at every time step. To avoid issues, check if the Slice contains the data you want before you index it. Each Factors object behaves like a dictionary, mapping the factor name to its value. Index the object with a factor name constant (for example, Factors.Technical.Roc1DayFactors.Technical.ROC_1_DAY is "technical_roc_1d") to get the value.

def on_data(self, data: Slice) -> None:
    equity_factors = data.get(Factors).get(self._equity.factors)
    if equity_factors:
        value = equity_factors[Factors.ROC_1_DAY]

    unlinked_factors = data.get(Factors).get(self._unlinked_factors)
    if unlinked_factors:
        ust_yield = unlinked_factors[Factors.UST_YIELD_1_MONTH]
public override void OnData(Slice data)
{
    if (data.Get<Factors>().TryGetValue(_equityFactors, out var equityFactors))
    {
        var value = equityFactors[Factors.Roc1Day];
    }

    if (data.Get<Factors>().TryGetValue(_unlinkedFactors, out var unlinkedFactors))
    {
        var ustYield = unlinkedFactors[Factors.UstYield1Month];
    }
}

Historical Data

To get historical factor data, call the Historyhistory method with the dataset Symbol. If there is no data in the period you request, the history result is empty.

# DataFrames
equity_factors_history_df = self.history(self._equity.factors, 100, Resolution.DAILY)
unlinked_factors_history_df = self.history(self._unlinked_factors, 100, Resolution.DAILY)

# Dataset objects
equity_factors_history = self.history[Factors](self._equity.factors, 100, Resolution.DAILY)
market_factors_history = self.history[Factors](self._unlinked_factors, 100, Resolution.DAILY)
var equityFactorsHistory = History<Factors>(_equityFactors, 100, Resolution.Daily);
var unlinkedFactorsHistory = History<Factors>(_marketFactors, 100, Resolution.Daily);

For more information about historical data, see History Requests.

Universe Selection

To select a dynamic universe of US Equities based on factor values, call the AddUniverseadd_universe method with the FactorsUniverse class and a selection function.

def initialize(self) -> None:
    self._universe = self.add_universe(FactorsUniverse, self._select_assets)

def _select_assets(self, data: List[FactorsUniverse]) -> List[Symbol]:
    ust_yield = None
    positive = []
    negative = []
    for factors in data:
        if Factors.UST_YIELD_1_MONTH in factors.keys:
            ust_yield = factors[Factors.UST_YIELD_1_MONTH]
        elif Factors.ROC_1_DAY in factors.keys:
            if factors[Factors.ROC_1_DAY] > 0:
                positive.append(factors.symbol)
            else:
                negative.append(factors.symbol)
    return positive if ust_yield < 4 else negative
private Universe _universe;

public override void Initialize()
{
    _universe = AddUniverse<FactorsUniverse>(SelectAssets);
}

private IEnumerable<Symbol> SelectAssets(IEnumerable<BaseData> data)
{
    decimal? ustYield = null;
    var positive = new List<Symbol>();
    var negative = new List<Symbol>();
    foreach (var factors in data.OfType<FactorsUniverse>())
    {
        if (factors.ContainsKey(Factors.UstYield1Month))
        {
            ustYield = factors[Factors.UstYield1Month];
        }
        else if (factors.ContainsKey(Factors.Roc1Day))
        {
            if (factors[Factors.Roc1Day] > 0)
            {
                positive.Add(factors.Symbol);
            }
            else
            {
                negative.Add(factors.Symbol);
            }
        }
    }
    return ustYield < 4 ? positive : negative;
}

For more information about dynamic universes, see Universes.

Universe History

You can get historical universe data in an algorithm and in the Research Environment.

Historical Universe Data in Algorithms

To get historical universe data in an algorithm, call the Historyhistory method with the Universe object and the lookback period. If there is no data in the period you request, the history result is empty.

var universeHistory = History(_universe, 30, Resolution.Daily);
foreach (var universeDay in universeHistory)
{
    foreach (FactorsUniverse factors in universeDay)
    {
        if (factors.TryGetValue(Factors.Roc1Day, out var value))
        {
            // ...
        }
    }
}
# DataFrame example where the columns are the FactorsUniverse attributes:
history_df = self.history(self._universe, 30, Resolution.DAILY, flatten=True)

# Series example where the values are lists of FactorsUniverse objects:
universe_history = self.history(self._universe, 30, Resolution.DAILY)
for (_, time), universe_day in universe_history.items():
    for factors in universe_day:
        if Factors.ROC_1_DAY in factors.keys:
            value = factors[Factors.ROC_1_DAY]

Historical Universe Data in Research

To get historical universe data in research, call the UniverseHistoryuniverse_history method with the Universe object, a start date, and an end date. This method returns the filtered universe. If there is no data in the period you request, the history result is empty.

var universeHistory = qb.History(universe, qb.Time.AddDays(-30), qb.Time);
foreach (var universeDay in universeHistory)
{
    foreach (FactorsUniverse factors in universeDay)
    {
        if (factors.TryGetValue(Factors.Roc1Day, out var value))
        {
            // ...
        }
    }
}
# DataFrame example where the columns are the FactorsUniverse attributes:
history = qb.history(universe, qb.time - timedelta(30), qb.time, flatten=True)

You can call the Historyhistory method in Research.

Remove Subscriptions

To remove a subscription, call the RemoveSecurityremove_security method.

self.remove_security(self._equity.factors)
RemoveSecurity(_equityFactors);

If you subscribe to Factors data for assets in a dynamic universe, remove the dataset subscription when the asset leaves your universe. To view a common design pattern, see Track Security Changes.

Supported Factors

The Factors and FactorsUniverse objects behave like a dictionary, mapping the factor name to its value. The Factors class provides name constants for every factor.

Linked Factors

Linked factors are factors that are tied to a specific tradable security. The Factors dataset provides the following linked factors for each US Equity:

  • Factors.ROC_1_DAYFactors.Roc1Day: 1-day rate of change of the closing price: close[t] / close[t-1] - 1
  • Factors.VOLATILITY_21_DAYFactors.Volatility21Day: 21-day realized volatility of daily returns: the standard deviation of 1-day closing-price returns over the trailing 21 days
  • Factors.VOLUME_Z_21_DAYFactors.VolumeZ21Day: 21-day volume z-score: the number of standard deviations the current trading volume sits above or below its trailing 21-day mean
  • Factors.MFI_21_DAYFactors.Mfi21Day: 21-day Money Flow Index: volume-weighted buying vs. selling pressure, from daily typical price changes signed and weighted by volume, accumulated over 21 sessions and bounded 0-100
  • Factors.MOM_12_MONTHFactors.Mom12Month: Twelve-month momentum skipping the most recent month: the stock's return over months t-11 through t-1, computed each session from LEAN's adjusted daily closes, from Jegadeesh and Titman (1993, JF), where higher values predicted higher returns
  • Factors.INT_MOMFactors.IntMom: Intermediate momentum: the stock's return over months t-12 through t-7, computed each session from LEAN's adjusted daily closes, from Novy-Marx (2012, JFE), where higher values predicted higher returns
  • Factors.MOM_12_MONTH_OFF_SEASONFactors.Mom12MonthOffSeason: Momentum without the seasonal month: the mean of the monthly returns of months t-1 through t-10, which leaves out the current month and month t-11, the month one year before the coming month, computed each session from LEAN's adjusted daily closes, from Heston and Sadka (2008, JFE), where higher values predicted higher returns
  • Factors.VOLUME_TRENDFactors.VolumeTrend: Volume trend: the slope of a regression of monthly volume on a linear time trend over the trailing 60 months, divided by the mean monthly volume over those months, computed each session from LEAN's split- and dividend-adjusted daily volume with 21 sessions to a month and at least 30 months required, from Haugen and Baker (1996, JFE), where higher values predicted lower returns
  • Factors.MOM_SEASON_SHORTFactors.MomSeasonShort: Return seasonality last year: the stock's return over the 21 sessions from 252 to 231 sessions ago, the month one year before the coming month, computed each session from LEAN's adjusted daily closes, from Heston and Sadka (2008, JFE), where higher values predicted higher returns
  • Factors.ILLIQUIDITYFactors.Illiquidity: Amihud illiquidity: the mean over the past 12 months of the daily ratio of the absolute return to the dollar volume traded, computed each session from LEAN's adjusted daily bars, from Amihud (2002, JFM), where higher values predicted higher returns
  • Factors.FINRA_SHORT_SALE_VOLUME_SHORT_VOLUME_RATIO_SMA_21_DAYFactors.FinraShortSaleVolumeShortVolumeRatioSma21Day: 21-day simple moving average of the FINRA Reg SHO daily short volume ratio: the share (0-1) of a security's off-exchange reported volume executed as short sales during regular trading hours
  • Factors.SPECTRAL_TICK_FLOW_SIGNAL_EXECUTION_SCOREFactors.SpectralTickFlowSignalExecutionScore: Normalized strength (non-negative) of the strongest recurring intraday execution rhythm in the security's daily tick flow, from a Fourier transform-like spectral decomposition of daily trades; higher means trade flow more strongly backed by an active, repeating execution signature
  • Factors.EARNING_YIELDFactors.EarningYield: Trailing earnings yield: Morningstar's ValuationRatios.EarningYield, diluted EPS divided by price (the inverse trailing P/E), so higher values indicate cheaper stocks relative to trailing earnings

Unlinked Factors

Unlinked factors are factors that aren't tied to a specific tradable security, like weather data or interest rates. The Factors dataset provides the following unlinked factors:

  • Factors.UST_YIELD_1_MONTHFactors.UstYield1Month: 1-month US Treasury yield curve rate, as published daily at 4:30 PM New York time
  • Factors.VIXFactors.Vix: CBOE Volatility Index (VIX) daily close: the market's expectation of 30-day S&P 500 volatility implied by index option prices, in annualized percentage points
  • Factors.WTI_CRUDE_OILFactors.WtiCrudeOil: WTI crude oil price: the daily mid close of Oanda's West Texas Oil CFD, in US dollars per barrel, a continuous quote that tracks the front NYMEX contract without roll jumps
  • Factors.MONETARY_CYCLEFactors.MonetaryCycle: Monetary cycle: the monetary environment of Jensen, Mercer and Johnson (1996), +1 while the most recent change in the Federal Reserve's primary credit rate was a decrease (expansive) and -1 while it was an increase (restrictive), read from LEAN's risk-free interest rate model for every calendar day at midnight New York; the state flips on the day the model's rate changes
  • Factors.M_2_GROWTH_1_MONTHFactors.M2Growth1Month: M2 money supply growth from one monthly release to the next: the Federal Reserve's seasonally adjusted monthly M2 over its value for the month before, minus one, from FRED's series M2SL as currently revised; each month's value is stamped at 13:00 New York on the fourth Tuesday of the following month, when the Federal Reserve's H.6 release publishes it

Example Applications

The QuantConnect Factors dataset enables researchers to accurately design strategies harnessing precomputed, point-in-time factor values. Examples include:

  • Factor momentum: weight each factor by its recent performance and form a long-short portfolio of the top-ranked stocks and bottom-ranked stocks
  • Momentum rotation: rank the universe by 1-day rate of change and hold the leaders
  • Rate regime filters: scale Equity exposure with the level of the 1-month US Treasury yield

Classic Algorithm Example

The following example algorithm ranks the S&P 500 constituents on every linked factor each day and weights each factor by how it performed over the trailing year in market conditions like today's, as measured by unlinked factors such as the VIX and the 1-month Treasury yield. It then buys the 5% of constituents with the highest weighted-rank scores, sizes each position by its score, and rebalances at every market open.

# region imports
from AlgorithmImports import *
from collections import deque
from datetime import date
from scipy.stats import rankdata
# endregion


class DaySnapshot:
    """What the algorithm knows about one trading day before its open."""

    def __init__(self, day: date, symbols: List[Symbol], ranks: np.ndarray, active: np.ndarray, unlinked: np.ndarray) -> None:
        self.day = day                # the trading day
        self.symbols = symbols        # S&P 500 constituents of the day
        self.ranks = ranks            # scaled ranks: one row per constituent, one column per linked factor
        self.active = active          # linked factors with ranks that day
        self.unlinked = unlinked      # unlinked factor values known before the open


class FactorsPortfolioAlgorithm(QCAlgorithm):
    """Conditional multi-factor long-only portfolio on the QuantConnect Factors dataset.

    Every trading day before the open, the algorithm ranks the S&P 500 constituents on each
    linked factor, measures each factor's daily payoff with a cross-sectional regression of
    open-to-open returns on all the ranks at once, and predicts each factor's payoff from its
    trailing-year payoffs, conditioned on the standardized unlinked factors. A stock's score is
    its predicted open-to-open return relative to the day's average. The algorithm holds the
    top 5% of the scores, weighted by score and fully invested.
    """

    _MONTH = 21                       # sessions in the dataset's month
    _LOOKBACK = 12 * _MONTH           # complete days in the estimation window
    _TAIL_FRACTION = 0.05             # share of the eligible stocks the portfolio holds

    def initialize(self) -> None:
        self.set_start_date(2024, 11, 1)
        self.set_end_date(2026, 10, 1)
        self.set_cash(10_000_000)
        self.settings.seed_initial_prices = True
        self.universe_settings.resolution = Resolution.DAILY
        self._linked = list(Factors.EQUITY_FACTOR_NAMES)
        self._unlinked = list(Factors.UNLINKED_FACTOR_NAMES)
        self._members = {}            # constituent -> linked factor values, from the latest universe data
        self._new_data = False        # whether universe data arrived since the last rebalance
        self._unlinked_values = {}    # most recent value of each unlinked factor
        self._opens = {}              # session date -> {symbol: open price}
        self._days = deque(maxlen=2)  # the last two trading days, kept until their returns are complete
        self._payoffs = deque(maxlen=self._LOOKBACK)  # (day, payoff of each linked factor, unlinked values)
        self._first_payoff = {}       # linked factor index -> day of its first payoff
        self.add_universe(FactorsUniverse, self._select)
        self.schedule.on(self.date_rules.every_day("SPY"), self.time_rules.at(8, 0), self._rebalance)
        # The estimation window comes from the year before the start date.
        self.set_warm_up(timedelta(days=380))

    def _select(self, rows: List[Factors]) -> List[Symbol]:
        previous = set(self._members)
        members = {}
        for row in rows:
            keys = set(row.keys)
            if row.is_unlinked:
                self._unlinked_values.update({name: float(row[name]) for name in self._unlinked if name in keys})
            else:
                members[row.symbol] = {name: float(row[name]) for name in self._linked if name in keys}
        self._members = members
        self._new_data = True
        # Keep the previous constituents subscribed one more session, so a stock that leaves the
        # index still has the next open that completes its last open-to-open return.
        return list(previous | set(members))

    def on_data(self, slice: Slice) -> None:
        for symbol, bar in slice.bars.items():
            if not bar.is_fill_forward:
                self._opens.setdefault(bar.time.date(), {})[symbol] = bar.open

    def _rebalance(self) -> None:
        # The day two sessions back now has a complete open-to-open return.
        if len(self._days) == 2:
            self._record_payoff(*self._days)
        today = self._snapshot()
        self._days.append(today)
        if self.is_warming_up or today.ranks is None:
            return
        model = self._factor_weights(today.unlinked)
        if model is None:
            return
        carried, weights = model
        targets = self._targets(today, carried, weights)
        names = [name for name, keep in zip(self._linked, carried) if keep]
        for name, weight in zip(names, weights):
            self.plot("Factor Weights", name, float(weight))
        self.set_holdings(targets, liquidate_existing_holdings=True)

    def _snapshot(self) -> DaySnapshot:
        unlinked = np.array([self._unlinked_values.get(name, np.nan) for name in self._unlinked])
        if not self._new_data:
            return DaySnapshot(self.time.date(), [], None, None, unlinked)
        self._new_data = False
        symbols = list(self._members)
        ranks = np.zeros((len(symbols), len(self._linked)))
        active = np.zeros(len(self._linked), dtype=bool)
        for k, name in enumerate(self._linked):
            rows = [i for i, symbol in enumerate(symbols) if name in self._members[symbol]]
            if len(rows) < 2:
                continue
            values = [self._members[symbols[i]][name] for i in rows]
            # Ranks scaled to run from -1/2 to 1/2; a constituent without a value keeps the middle rank, 0.
            ranks[rows, k] = (rankdata(values) - 1) / (len(rows) - 1) - 0.5
            active[k] = True
        if not active.any():
            return DaySnapshot(self.time.date(), symbols, None, None, unlinked)
        return DaySnapshot(self.time.date(), symbols, ranks, active, unlinked)

    def _record_payoff(self, day: DaySnapshot, next_day: DaySnapshot) -> None:
        start_opens = self._opens.pop(day.day, {})
        end_opens = self._opens.get(next_day.day, {})
        for stale in [session for session in self._opens if session < next_day.day]:
            del self._opens[stale]
        if day.ranks is None:
            return
        rows, returns = [], []
        for i, symbol in enumerate(day.symbols):
            start, end = start_opens.get(symbol), end_opens.get(symbol)
            if start and end:
                rows.append(i)
                returns.append(end / start - 1)
        if not rows:
            return
        # Regress the open-to-open returns on all active factors' ranks at once, with an intercept.
        columns = np.flatnonzero(day.active)
        design = np.column_stack([np.ones(len(rows)), day.ranks[np.ix_(rows, columns)]])
        coefficients = np.linalg.lstsq(design, np.array(returns), rcond=None)[0]
        payoff = np.full(len(self._linked), np.nan)
        payoff[columns] = coefficients[1:]
        self._payoffs.append((day.day, payoff, day.unlinked))
        for k in columns:
            self._first_payoff.setdefault(k, day.day)

    def _factor_weights(self, unlinked_today: np.ndarray) -> tuple[np.ndarray, np.ndarray] | None:
        """The linked factors that carry weight today, and the predicted payoff of each."""
        if len(self._payoffs) < self._LOOKBACK:
            return None
        window_start = self._payoffs[0][0]
        # A linked factor carries weight once its payoff history spans the full window.
        carried = np.array([k in self._first_payoff and self._first_payoff[k] <= window_start for k in range(len(self._linked))])
        if not carried.any():
            return None
        payoffs = np.array([payoff for _, payoff, _ in self._payoffs])[:, carried]
        unlinked = np.array([values for _, _, values in self._payoffs])
        rows = ~np.isnan(payoffs).any(axis=1)
        # An unlinked factor takes part when it has values throughout the window and today, and it moved.
        known = ~np.isnan(unlinked).any(axis=0) & ~np.isnan(unlinked_today)
        mean = unlinked[:, known].mean(axis=0)
        std = unlinked[:, known].std(axis=0)
        moved = std > 0
        z = (unlinked[:, known][:, moved] - mean[moved]) / std[moved]
        z_today = (unlinked_today[known][moved] - mean[moved]) / std[moved]
        design = np.column_stack([np.ones(rows.sum()), z[rows]])
        # The fit needs more complete days than coefficients.
        if len(design) <= design.shape[1]:
            return None
        # Regress each factor's payoffs on the standardized unlinked values and predict at today's values.
        coefficients = np.linalg.lstsq(design, payoffs[rows], rcond=None)[0]
        point = np.concatenate([[1.0], z_today])
        return carried, point @ coefficients

    def _targets(self, today: DaySnapshot, carried: np.ndarray, weights: np.ndarray) -> List[PortfolioTarget]:
        eligible = [
            i for i, symbol in enumerate(today.symbols)
            if symbol in self.securities and self.securities[symbol].is_tradable and self.securities[symbol].price > 0
        ]
        if not eligible:
            return []
        scores = today.ranks[eligible][:, carried] @ weights
        # Hold the highest tail of the scores, each stock weighted by its score above the day's average.
        excess = scores - scores.mean()
        count = int(len(eligible) * self._TAIL_FRACTION)
        top = np.argsort(excess)[len(eligible) - count:]
        positions = np.zeros(len(eligible))
        positions[top] = np.maximum(excess[top], 0)
        total = positions.sum()
        if total == 0:
            return []
        positions = positions / total
        return [PortfolioTarget(today.symbols[i], weight) for i, weight in zip(eligible, positions) if weight != 0]
#region imports
using System;
using System.Collections.Generic;
using System.Linq;
using MathNet.Numerics.LinearAlgebra;
using QuantConnect.Algorithm.Framework.Portfolio;
using QuantConnect.Data;
using QuantConnect.DataSource;
#endregion

namespace QuantConnect.Algorithm.CSharp
{
    /// <summary>
    /// What the algorithm knows about one trading day before its open.
    /// </summary>
    public class DaySnapshot
    {
        public DateTime Day;          // the trading day
        public List<Symbol> Symbols;  // S&P 500 constituents of the day
        public double[,] Ranks;       // scaled ranks: one row per constituent, one column per linked factor
        public bool[] Active;         // linked factors with ranks that day
        public double[] Unlinked;     // unlinked factor values known before the open
    }

    /// <summary>
    /// Conditional multi-factor long-only portfolio on the QuantConnect Factors dataset.
    ///
    /// Every trading day before the open, the algorithm ranks the S&P 500 constituents on each
    /// linked factor, measures each factor's daily payoff with a cross-sectional regression of
    /// open-to-open returns on all the ranks at once, and predicts each factor's payoff from its
    /// trailing-year payoffs, conditioned on the standardized unlinked factors. A stock's score is
    /// its predicted open-to-open return relative to the day's average. The algorithm holds the
    /// top 5% of the scores, weighted by score and fully invested.
    /// </summary>
    public class FactorsPortfolioAlgorithm : QCAlgorithm
    {
        private const int Month = 21;               // sessions in the dataset's month
        private const int Lookback = 12 * Month;    // complete days in the estimation window
        private const double TailFraction = 0.05;   // share of the eligible stocks the portfolio holds

        private string[] _linked;
        private string[] _unlinked;
        // constituent -> linked factor values, from the latest universe data
        private Dictionary<Symbol, Dictionary<string, double>> _members = new();
        // the constituents in the universe data's order
        private List<Symbol> _memberOrder = new();
        // whether universe data arrived since the last rebalance
        private bool _newData;
        // most recent value of each unlinked factor
        private readonly Dictionary<string, double> _unlinkedValues = new();
        // session date -> {symbol: open price}
        private readonly Dictionary<DateTime, Dictionary<Symbol, decimal>> _opens = new();
        // the last two trading days, kept until their returns are complete
        private readonly List<DaySnapshot> _days = new();
        // (day, payoff of each linked factor, unlinked values) of the Lookback most recent complete days
        private readonly List<(DateTime Day, double[] Payoff, double[] Unlinked)> _payoffs = new();
        // linked factor index -> day of its first payoff
        private readonly Dictionary<int, DateTime> _firstPayoff = new();

        public override void Initialize()
        {
            SetStartDate(2024, 11, 1);
            SetEndDate(2026, 10, 1);
            SetCash(10_000_000);
            Settings.SeedInitialPrices = true;
            UniverseSettings.Resolution = Resolution.Daily;
            _linked = Factors.EquityFactorNames;
            _unlinked = Factors.UnlinkedFactorNames;
            AddUniverse<FactorsUniverse>(Select);
            Schedule.On(DateRules.EveryDay("SPY"), TimeRules.At(8, 0), Rebalance);
            // The estimation window comes from the year before the start date.
            SetWarmUp(TimeSpan.FromDays(380));
        }

        private IEnumerable<Symbol> Select(IEnumerable<BaseData> rows)
        {
            var selected = _members.Keys.ToHashSet();
            var members = new Dictionary<Symbol, Dictionary<string, double>>();
            var order = new List<Symbol>();
            foreach (var row in rows.OfType<Factors>())
            {
                if (row.IsUnlinked)
                {
                    foreach (var name in _unlinked)
                    {
                        if (row.TryGetValue(name, out var value))
                        {
                            _unlinkedValues[name] = (double)value;
                        }
                    }
                    continue;
                }
                var values = new Dictionary<string, double>();
                foreach (var name in _linked)
                {
                    if (row.TryGetValue(name, out var value))
                    {
                        values[name] = (double)value;
                    }
                }
                if (members.TryAdd(row.Symbol, values))
                {
                    order.Add(row.Symbol);
                }
            }
            _members = members;
            _memberOrder = order;
            _newData = true;
            // Keep the previous constituents subscribed one more session, so a stock that leaves the
            // index still has the next open that completes its last open-to-open return.
            selected.UnionWith(order);
            return selected;
        }

        public override void OnData(Slice slice)
        {
            foreach (var bar in slice.Bars.Values)
            {
                if (bar.IsFillForward)
                {
                    continue;
                }
                if (!_opens.TryGetValue(bar.Time.Date, out var opens))
                {
                    _opens[bar.Time.Date] = opens = new Dictionary<Symbol, decimal>();
                }
                opens[bar.Symbol] = bar.Open;
            }
        }

        private void Rebalance()
        {
            // The day two sessions back now has a complete open-to-open return.
            if (_days.Count == 2)
            {
                RecordPayoff(_days[0], _days[1]);
                _days.RemoveAt(0);
            }
            var today = Snapshot();
            _days.Add(today);
            if (IsWarmingUp || today.Ranks == null)
            {
                return;
            }
            var model = FactorWeights(today.Unlinked);
            if (model == null)
            {
                return;
            }
            var (carried, weights) = model.Value;
            var targets = Targets(today, carried, weights);
            var names = _linked.Where((name, k) => carried[k]).ToArray();
            for (var j = 0; j < names.Length; j++)
            {
                Plot("Factor Weights", names[j], weights[j]);
            }
            SetHoldings(targets, liquidateExistingHoldings: true);
        }

        private DaySnapshot Snapshot()
        {
            var unlinked = _unlinked
                .Select(name => _unlinkedValues.TryGetValue(name, out var value) ? value : double.NaN)
                .ToArray();
            if (!_newData)
            {
                return new DaySnapshot { Day = Time.Date, Symbols = new List<Symbol>(), Unlinked = unlinked };
            }
            _newData = false;
            var symbols = _memberOrder;
            var ranks = new double[symbols.Count, _linked.Length];
            var active = new bool[_linked.Length];
            for (var k = 0; k < _linked.Length; k++)
            {
                var name = _linked[k];
                var rows = Enumerable.Range(0, symbols.Count)
                    .Where(i => _members[symbols[i]].ContainsKey(name))
                    .ToList();
                if (rows.Count < 2)
                {
                    continue;
                }
                var rank = AverageRanks(rows.Select(i => _members[symbols[i]][name]).ToArray());
                // Ranks scaled to run from -1/2 to 1/2; a constituent without a value keeps the middle rank, 0.
                for (var j = 0; j < rows.Count; j++)
                {
                    ranks[rows[j], k] = (rank[j] - 1) / (rows.Count - 1) - 0.5;
                }
                active[k] = true;
            }
            if (!active.Any(isActive => isActive))
            {
                return new DaySnapshot { Day = Time.Date, Symbols = symbols, Unlinked = unlinked };
            }
            return new DaySnapshot { Day = Time.Date, Symbols = symbols, Ranks = ranks, Active = active, Unlinked = unlinked };
        }

        // Ranks from 1 for the lowest value to n for the highest; tied values share their average rank.
        private static double[] AverageRanks(double[] values)
        {
            var order = Enumerable.Range(0, values.Length).OrderBy(i => values[i]).ToArray();
            var ranks = new double[values.Length];
            var start = 0;
            while (start < order.Length)
            {
                var end = start;
                while (end + 1 < order.Length && values[order[end + 1]] == values[order[start]])
                {
                    end++;
                }
                for (var j = start; j <= end; j++)
                {
                    ranks[order[j]] = (start + end) / 2.0 + 1;
                }
                start = end + 1;
            }
            return ranks;
        }

        private void RecordPayoff(DaySnapshot day, DaySnapshot nextDay)
        {
            _opens.Remove(day.Day, out var startOpens);
            _opens.TryGetValue(nextDay.Day, out var endOpens);
            foreach (var stale in _opens.Keys.Where(session => session < nextDay.Day).ToList())
            {
                _opens.Remove(stale);
            }
            if (day.Ranks == null || startOpens == null || endOpens == null)
            {
                return;
            }
            var rows = new List<int>();
            var returns = new List<double>();
            for (var i = 0; i < day.Symbols.Count; i++)
            {
                if (startOpens.TryGetValue(day.Symbols[i], out var start) && start != 0
                    && endOpens.TryGetValue(day.Symbols[i], out var end) && end != 0)
                {
                    rows.Add(i);
                    returns.Add((double)end / (double)start - 1);
                }
            }
            if (rows.Count == 0)
            {
                return;
            }
            // Regress the open-to-open returns on all active factors' ranks at once, with an intercept.
            var columns = Enumerable.Range(0, _linked.Length).Where(k => day.Active[k]).ToArray();
            var design = Matrix<double>.Build.Dense(rows.Count, columns.Length + 1,
                (i, j) => j == 0 ? 1 : day.Ranks[rows[i], columns[j - 1]]);
            var coefficients = design.PseudoInverse() * Vector<double>.Build.DenseOfEnumerable(returns);
            var payoff = Enumerable.Repeat(double.NaN, _linked.Length).ToArray();
            for (var j = 0; j < columns.Length; j++)
            {
                payoff[columns[j]] = coefficients[j + 1];
                _firstPayoff.TryAdd(columns[j], day.Day);
            }
            _payoffs.Add((day.Day, payoff, day.Unlinked));
            if (_payoffs.Count > Lookback)
            {
                _payoffs.RemoveAt(0);
            }
        }

        // The linked factors that carry weight today, and the predicted payoff of each.
        private (bool[] Carried, double[] Weights)? FactorWeights(double[] unlinkedToday)
        {
            if (_payoffs.Count < Lookback)
            {
                return null;
            }
            var windowStart = _payoffs[0].Day;
            // A linked factor carries weight once its payoff history spans the full window.
            var carried = Enumerable.Range(0, _linked.Length)
                .Select(k => _firstPayoff.TryGetValue(k, out var first) && first <= windowStart)
                .ToArray();
            var factors = Enumerable.Range(0, _linked.Length).Where(k => carried[k]).ToArray();
            if (factors.Length == 0)
            {
                return null;
            }
            var rows = _payoffs.Where(day => factors.All(k => !double.IsNaN(day.Payoff[k]))).ToList();
            // An unlinked factor takes part when it has values throughout the window and today, and it moved.
            var conditions = new List<(int Index, double Mean, double Std)>();
            for (var j = 0; j < _unlinked.Length; j++)
            {
                var index = j;
                if (double.IsNaN(unlinkedToday[index]) || _payoffs.Any(day => double.IsNaN(day.Unlinked[index])))
                {
                    continue;
                }
                var mean = _payoffs.Average(day => day.Unlinked[index]);
                var std = Math.Sqrt(_payoffs.Average(day => Math.Pow(day.Unlinked[index] - mean, 2)));
                if (std > 0)
                {
                    conditions.Add((index, mean, std));
                }
            }
            var design = Matrix<double>.Build.Dense(rows.Count, conditions.Count + 1,
                (i, j) => j == 0 ? 1 : (rows[i].Unlinked[conditions[j - 1].Index] - conditions[j - 1].Mean) / conditions[j - 1].Std);
            // The fit needs more complete days than coefficients.
            if (design.RowCount <= design.ColumnCount)
            {
                return null;
            }
            // Regress each factor's payoffs on the standardized unlinked values and predict at today's values.
            var payoffs = Matrix<double>.Build.Dense(rows.Count, factors.Length, (i, j) => rows[i].Payoff[factors[j]]);
            var coefficients = design.PseudoInverse() * payoffs;
            var point = Vector<double>.Build.Dense(conditions.Count + 1,
                j => j == 0 ? 1 : (unlinkedToday[conditions[j - 1].Index] - conditions[j - 1].Mean) / conditions[j - 1].Std);
            return (carried, (coefficients.Transpose() * point).ToArray());
        }

        private List<PortfolioTarget> Targets(DaySnapshot today, bool[] carried, double[] weights)
        {
            var eligible = Enumerable.Range(0, today.Symbols.Count)
                .Where(i => Securities.TryGetValue(today.Symbols[i], out var security) && security.IsTradable && security.Price > 0)
                .ToList();
            if (eligible.Count == 0)
            {
                return new List<PortfolioTarget>();
            }
            var factors = Enumerable.Range(0, _linked.Length).Where(k => carried[k]).ToArray();
            var scores = eligible
                .Select(i => factors.Select((k, j) => today.Ranks[i, k] * weights[j]).Sum())
                .ToArray();
            // Hold the highest tail of the scores, each stock weighted by its score above the day's average.
            var mean = scores.Average();
            var excess = scores.Select(score => score - mean).ToArray();
            var count = (int)(eligible.Count * TailFraction);
            var top = Enumerable.Range(0, eligible.Count)
                .OrderBy(i => excess[i])
                .Skip(eligible.Count - count)
                .ToHashSet();
            var positions = excess.Select((value, i) => top.Contains(i) ? Math.Max(value, 0) : 0).ToArray();
            var total = positions.Sum();
            if (total == 0)
            {
                return new List<PortfolioTarget>();
            }
            return eligible
                .Select((i, j) => (Symbol: today.Symbols[i], Weight: positions[j] / total))
                .Where(target => target.Weight != 0)
                .Select(target => new PortfolioTarget(target.Symbol, (decimal)target.Weight))
                .ToList();
        }
    }
}

Framework Algorithm Example

The following example algorithm selects the SPY constituents, the dataset's coverage universe. The Alpha model emits daily ROC-weighted insights for the Equities that moved more than 1% over the last trading session, targeting only long positions while the 1-month US Treasury yield is below 4% and only short positions otherwise.

from AlgorithmImports import *


class FactorsFrameworkAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2025, 9, 1)
        self.set_end_date(2025, 12, 31)
        self.set_cash(10_000_000)

        self.universe_settings.resolution = Resolution.DAILY
        self.add_universe_selection(ETFConstituentsUniverseSelectionModel("SPY"))
        self.add_alpha(FactorsAlphaModel(self))
        self.set_portfolio_construction(InsightWeightingPortfolioConstructionModel())


class FactorsAlphaModel(AlphaModel):

    def __init__(self, algorithm: QCAlgorithm) -> None:
        self._unlinked_factors = algorithm.add_data(Factors, Factors.UNLINKED).symbol
        self._roc_by_symbol = {}

    def update(self, algorithm: QCAlgorithm, data: Slice) -> List[Insight]:
        # Cache the Equities that moved more than 1%. Their factors arrive with
        # the session's 4PM close, ahead of the unlinked factors row.
        for symbol, factors in data.get(Factors).items():
            if Factors.ROC_1_DAY in factors.keys and abs(factors[Factors.ROC_1_DAY]) > 0.01:
                self._roc_by_symbol[symbol.underlying] = factors[Factors.ROC_1_DAY]

        # The unlinked factors arrive at 4:30PM, when the US Treasury publishes
        # the daily yield curve rates, completing the session's signals.
        unlinked_factors = data.get(Factors).get(self._unlinked_factors)
        if not unlinked_factors or Factors.UST_YIELD_1_MONTH not in unlinked_factors.keys:
            return []

        # Consume the session's cache so stale values never roll into the
        # next emission.
        roc_by_symbol = self._roc_by_symbol
        self._roc_by_symbol = {}

        # While short-term rates are low, only target long assets. Otherwise,
        # only target short assets.
        if unlinked_factors[Factors.UST_YIELD_1_MONTH] < 4:
            direction = InsightDirection.UP
            selected = {s: roc for s, roc in roc_by_symbol.items() if roc > 0}
        else:
            direction = InsightDirection.DOWN
            selected = {s: roc for s, roc in roc_by_symbol.items() if roc < 0}
        if not selected:
            return []

        # Weight each insight by the magnitude of the asset's ROC.
        total_roc = sum(abs(roc) for roc in selected.values())
        return [
            Insight.price(symbol, timedelta(1), direction, weight=abs(roc)/total_roc)
            for symbol, roc in selected.items()
        ]

    def on_securities_changed(self, algorithm: QCAlgorithm, changes: SecurityChanges) -> None:
        for security in changes.added_securities:
            security.factors = algorithm.add_data(Factors, security.symbol).symbol
        for security in changes.removed_securities:
            algorithm.remove_security(security.factors)
            self._roc_by_symbol.pop(security.symbol, None)
public class FactorsFrameworkAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        SetStartDate(2025, 9, 1);
        SetEndDate(2025, 12, 31);
        SetCash(10000000);

        UniverseSettings.Resolution = Resolution.Daily;
        AddUniverseSelection(new ETFConstituentsUniverseSelectionModel("SPY"));
        AddAlpha(new FactorsAlphaModel(this));
        SetPortfolioConstruction(new InsightWeightingPortfolioConstructionModel());
    }
}

public class FactorsAlphaModel : AlphaModel
{
    private readonly Symbol _unlinkedFactors;
    private Dictionary<Symbol, decimal> _rocBySymbol = new Dictionary<Symbol, decimal>();

    public FactorsAlphaModel(QCAlgorithm algorithm)
    {
        _unlinkedFactors = algorithm.AddData<Factors>(Factors.Unlinked).Symbol;
    }

    public override IEnumerable<Insight> Update(QCAlgorithm algorithm, Slice data)
    {
        // Cache the Equities that moved more than 1%. Their factors arrive with
        // the session's 4PM close, ahead of the unlinked factors row.
        foreach (var factors in data.Get<Factors>().Values)
        {
            if (factors.ContainsKey(Factors.Roc1Day) && Math.Abs(factors[Factors.Roc1Day]) > 0.01m)
            {
                _rocBySymbol[factors.Symbol.Underlying] = factors[Factors.Roc1Day];
            }
        }

        // The unlinked factors arrive at 4:30PM, when the US Treasury publishes
        // the daily yield curve rates, completing the session's signals.
        if (!data.Get<Factors>().TryGetValue(_unlinkedFactors, out var unlinkedFactors) || !unlinkedFactors.ContainsKey(Factors.UstYield1Month))
        {
            return Enumerable.Empty<Insight>();
        }

        // Consume the session's cache so stale values never roll into the
        // next emission.
        var rocBySymbol = _rocBySymbol;
        _rocBySymbol = new Dictionary<Symbol, decimal>();

        // While short-term rates are low, only target long assets. Otherwise,
        // only target short assets.
        InsightDirection direction;
        List<KeyValuePair<Symbol, decimal>> selected;
        if (unlinkedFactors[Factors.UstYield1Month] < 4)
        {
            direction = InsightDirection.Up;
            selected = rocBySymbol.Where(kvp => kvp.Value > 0).ToList();
        }
        else
        {
            direction = InsightDirection.Down;
            selected = rocBySymbol.Where(kvp => kvp.Value < 0).ToList();
        }
        if (selected.Count == 0)
        {
            return Enumerable.Empty<Insight>();
        }

        // Weight each insight by the magnitude of the asset's ROC.
        var totalRoc = selected.Sum(kvp => Math.Abs(kvp.Value));
        return selected.Select(kvp =>
            Insight.Price(kvp.Key, TimeSpan.FromDays(1), direction, weight: (double)(Math.Abs(kvp.Value) / totalRoc)));
    }

    public override void OnSecuritiesChanged(QCAlgorithm algorithm, SecurityChanges changes)
    {
        foreach (dynamic security in changes.AddedSecurities)
        {
            security.Factors = algorithm.AddData<Factors>(security.Symbol).Symbol;
        }
        foreach (dynamic security in changes.RemovedSecurities)
        {
            algorithm.RemoveSecurity(security.Factors);
            _rocBySymbol.Remove(security.Symbol);
        }
    }
}

Data Point Attributes

The Factors dataset provides Factors and FactorsUniverse objects.

Factors Attributes

Factors objects has the following attributes:

FactorsUniverse Attributes

FactorsUniverse objects has the following attributes:

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: