Datasets

Equity

Introduction

This page explains how to request, manipulate, and visualize historical US Equity data.

Prerequisites

Working knowledge of C#.

Working knowedge of Python and pandas. If you are not familiar with pandas, see the pandas documentation.

Create Subscriptions

Follow these steps to subscribe to an Equity security:

  1. Load the required assembly files and data types.
  2. #load "../Initialize.csx"
    #load "../QuantConnect.csx"
    
    using QuantConnect;
    using QuantConnect.Data;
    using QuantConnect.Algorithm;
    using QuantConnect.Research;
    using QuantConnect.Indicators;
  3. Create a QuantBook.
  4. var qb = new QuantBook();
    qb = QuantBook()
  5. Call the AddEquity method with a ticker and then save a reference to the Equity Symbol.
  6. var spy = qb.AddEquity("SPY").Symbol;
    var tlt = qb.AddEquity("TLT").Symbol;
    spy = qb.AddEquity("SPY").Symbol
    tlt = qb.AddEquity("TLT").Symbol

To view the supported assets in the US Equities dataset, see the Data Explorer.

Get Historical Data

You need a subscription before you can request historical data for a security. On the time dimension, you can request an amount of historical data based on a trailing number of bars, a trailing period of time, or a defined period of time. On the security dimension, you can request historical data for a single Equity, a subset of the Equities you created subscriptions for in your notebook, or all of the Equities in your notebook.

Trailing Number of Bars

To get historical data for a number of trailing bars, call the History method with the Symbol object(s) and an integer.

// Slice objects
var singleHistorySlice = qb.History(spy, 10);
var subsetHistorySlice = qb.History(new[] {spy, tlt}, 10);
var allHistorySlice = qb.History(10);

// TradeBar objects
var singleHistoryTradeBars = qb.History<TradeBar>(spy, 10);
var subsetHistoryTradeBars = qb.History<TradeBar>(new[] {spy, tlt}, 10);
var allHistoryTradeBars = qb.History<TradeBar>(qb.Securities.Keys, 10);

// QuoteBar objects
var singleHistoryQuoteBars = qb.History<QuoteBar>(spy, 10);
var subsetHistoryQuoteBars = qb.History<QuoteBar>(new[] {spy, tlt}, 10);
var allHistoryQuoteBars = qb.History<QuoteBar>(qb.Securities.Keys, 10);
# DataFrame of trade and quote data
single_history_df = qb.History(spy, 10)
subset_history_df = qb.History([spy, tlt], 10)
all_history_df = qb.History(qb.Securities.Keys, 10)

# DataFrame of trade data
single_history_trade_bar_df = qb.History(TradeBar, spy, 10)
subset_history_trade_bar_df = qb.History(TradeBar, [spy, tlt], 10)
all_history_trade_bar_df = qb.History(TradeBar, qb.Securities.Keys, 10)

# DataFrame of quote data
single_history_quote_bar_df = qb.History(QuoteBar, spy, 10)
subset_history_quote_bar_df = qb.History(QuoteBar, [spy, tlt], 10)
all_history_quote_bar_df = qb.History(QuoteBar, qb.Securities.Keys, 10)

# Slice objects
all_history_slice = qb.History(10)

# TradeBar objects
single_history_trade_bars = qb.History[TradeBar](spy, 10)
subset_history_trade_bars = qb.History[TradeBar]([spy, tlt], 10)
all_history_trade_bars = qb.History[TradeBar](qb.Securities.Keys, 10)

# QuoteBar objects
single_history_quote_bars = qb.History[QuoteBar](spy, 10)
subset_history_quote_bars = qb.History[QuoteBar]([spy, tlt], 10)
all_history_quote_bars = qb.History[QuoteBar](qb.Securities.Keys, 10)

The preceding calls return the most recent bars, excluding periods of time when the exchange was closed.

Trailing Period of Time

To get historical data for a trailing period of time, call the History method with the Symbol object(s) and a TimeSpantimedelta.

// Slice objects
var singleHistorySlice = qb.History(spy, TimeSpan.FromDays(3));
var subsetHistorySlice = qb.History(new[] {spy, tlt}, TimeSpan.FromDays(3));
var allHistorySlice = qb.History(10);

// TradeBar objects
var singleHistoryTradeBars = qb.History<TradeBar>(spy, TimeSpan.FromDays(3));
var subsetHistoryTradeBars = qb.History<TradeBar>(new[] {spy, tlt}, TimeSpan.FromDays(3));
var allHistoryTradeBars = qb.History<TradeBar>(TimeSpan.FromDays(3));

// QuoteBar objects
var singleHistoryQuoteBars = qb.History<QuoteBar>(spy, TimeSpan.FromDays(3), Resolution.Minute);
var subsetHistoryQuoteBars = qb.History<QuoteBar>(new[] {spy, tlt}, TimeSpan.FromDays(3), Resolution.Minute);
var allHistoryQuoteBars = qb.History<QuoteBar>(qb.Securities.Keys, TimeSpan.FromDays(3), Resolution.Minute);

// Tick objects
var singleHistoryTicks = qb.History<Tick>(spy, TimeSpan.FromDays(3), Resolution.Tick);
var subsetHistoryTicks = qb.History<Tick>(new[] {spy, tlt}, TimeSpan.FromDays(3), Resolution.Tick);
var allHistoryTicks = qb.History<Tick>(qb.Securities.Keys, TimeSpan.FromDays(3), Resolution.Tick);
# DataFrame of trade and quote data
single_history_df = qb.History(spy, timedelta(days=3))
subset_history_df = qb.History([spy, tlt], timedelta(days=3))
all_history_df = qb.History(qb.Securities.Keys, timedelta(days=3))

# DataFrame of trade data
single_history_trade_bar_df = qb.History(TradeBar, spy, timedelta(days=3))
subset_history_trade_bar_df = qb.History(TradeBar, [spy, tlt], timedelta(days=3))
all_history_trade_bar_df = qb.History(TradeBar, qb.Securities.Keys, timedelta(days=3))

# DataFrame of quote data
single_history_quote_bar_df = qb.History(QuoteBar, spy, timedelta(days=3))
subset_history_quote_bar_df = qb.History(QuoteBar, [spy, tlt], timedelta(days=3))
all_history_quote_bar_df = qb.History(QuoteBar, qb.Securities.Keys, timedelta(days=3))

# Slice objects
all_history_slice = qb.History(timedelta(days=3))

# TradeBar objects
single_history_trade_bars = qb.History[TradeBar](spy, timedelta(days=3))
subset_history_trade_bars = qb.History[TradeBar]([spy, tlt], timedelta(days=3))
all_history_trade_bars = qb.History[TradeBar](qb.Securities.Keys, timedelta(days=3))

# QuoteBar objects
single_history_quote_bars = qb.History[QuoteBar](spy, timedelta(days=3), Resolution.Minute)
subset_history_quote_bars = qb.History[QuoteBar]([spy, tlt], timedelta(days=3), Resolution.Minute)
all_history_quote_bars = qb.History[QuoteBar](qb.Securities.Keys, timedelta(days=3), Resolution.Minute)

# Tick objects
single_history_ticks = qb.History[Tick](spy, timedelta(days=3), Resolution.Tick)
subset_history_ticks = qb.History[Tick]([spy, tlt], timedelta(days=3), Resolution.Tick)
all_history_ticks = qb.History[Tick](qb.Securities.Keys, timedelta(days=3), Resolution.Tick)

The preceding calls return the most recent bars or ticks, excluding periods of time when the exchange was closed.

Defined Period of Time

To get historical data for a specific period of time, call the History method with the Symbol object(s), a start DateTimedatetime, and an end DateTimedatetime. The start and end times you provide are based in the notebook time zone.

var startTime = new DateTime(2021, 1, 1);
var endTime = new DateTime(2021, 2, 1);

// Slice objects
var singleHistorySlice = qb.History(spy, startTime, endTime);
var subsetHistorySlice = qb.History(new[] {spy, tlt}, startTime, endTime);
var allHistorySlice = qb.History(qb.Securities.Keys, startTime, endTime);

// TradeBar objects
var singleHistoryTradeBars = qb.History<TradeBar>(spy, startTime, endTime);
var subsetHistoryTradeBars = qb.History<TradeBar>(new[] {spy, tlt}, startTime, endTime);
var allHistoryTradeBars = qb.History<TradeBar>(qb.Securities.Keys, startTime, endTime);

// QuoteBar objects
var singleHistoryQuoteBars = qb.History<QuoteBar>(spy, startTime, endTime, Resolution.Minute);
var subsetHistoryQuoteBars = qb.History<QuoteBar>(new[] {spy, tlt}, startTime, endTime, Resolution.Minute);
var allHistoryQuoteBars = qb.History<QuoteBar>(qb.Securities.Keys, startTime, endTime, Resolution.Minute);

// Tick objects
var singleHistoryTicks = qb.History<Tick>(spy, startTime, endTime, Resolution.Tick);
var subsetHistoryTicks = qb.History<Tick>(new[] {spy, tlt}, startTime, endTime, Resolution.Tick);
var allHistoryTicks = qb.History<Tick>(qb.Securities.Keys, startTime, endTime, Resolution.Tick);
start_time = datetime(2021, 1, 1)
end_time = datetime(2021, 2, 1)

# DataFrame of trade and quote data
single_history_df = qb.History(spy, start_time, end_time)
subset_history_df = qb.History([spy, tlt], start_time, end_time)
all_history_df = qb.History(qb.Securities.Keys, start_time, end_time)

# DataFrame of trade data
single_history_trade_bar_df = qb.History(TradeBar, spy, start_time, end_time)
subset_history_trade_bar_df = qb.History(TradeBar, [spy, tlt], start_time, end_time)
all_history_trade_bar_df = qb.History(TradeBar, qb.Securities.Keys, start_time, end_time)

# DataFrame of quote data
single_history_quote_bar_df = qb.History(QuoteBar, spy, start_time, end_time)
subset_history_quote_bar_df = qb.History(QuoteBar, [spy, tlt], start_time, end_time)
all_history_quote_bar_df = qb.History(QuoteBar, qb.Securities.Keys, start_time, end_time)

# TradeBar objects
single_history_trade_bars = qb.History[TradeBar](spy, start_time, end_time)
subset_history_trade_bars = qb.History[TradeBar]([spy, tlt], start_time, end_time)
all_history_trade_bars = qb.History[TradeBar](qb.Securities.Keys, start_time, end_time)

# QuoteBar objects
single_history_quote_bars = qb.History[QuoteBar](spy, start_time, end_time, Resolution.Minute)
subset_history_quote_bars = qb.History[QuoteBar]([spy, tlt], start_time, end_time, Resolution.Minute)
all_history_quote_bars = qb.History[QuoteBar](qb.Securities.Keys, start_time, end_time, Resolution.Minute)

# Tick objects
single_history_ticks = qb.History[Tick](spy, start_time, end_time, Resolution.Tick)
subset_history_ticks = qb.History[Tick]([spy, tlt], start_time, end_time, Resolution.Tick)
all_history_ticks = qb.History[Tick](qb.Securities.Keys, start_time, end_time, Resolution.Tick)

The preceding calls return the bars or ticks that have a timestamp within the defined period of time.

Resolutions

The following table shows the available resolutions and data formats for Equity subscriptions:

ResolutionTradeBarQuoteBarTrade TickQuote Tick
Tickgreen checkgreen check
Secondgreen checkgreen check
Minutegreen checkgreen check
Hourgreen check
Dailygreen check

Markets

LEAN groups all of the US Equity exchanges under Market.USA.

Data Normalization

The data normalization mode defines how historical data is adjusted for corporate actions. By default, LEAN adjusts US Equity data for splits and dividends to produce a smooth price curve, but the following data normalization modes are available:

We use the entire split and dividend history to adjust historical prices. This process ensures you get the same adjusted prices, regardless of the QuantBook time.

To set the data normalization mode for a security, pass a dataNormalizationMode argument to the AddEquity method.

var spy = qb.AddEquity("SPY", dataNormalizationMode: DataNormalizationMode.Raw).Symbol;
spy = qb.AddEquity("SPY", dataNormalizationMode=DataNormalizationMode.Raw).Symbol

When you request historical data, the History method uses the data normalization of your security subscription. To get historical data with a different data normalization, pass a dataNormalizationMode argument to the History method.

var history = qb.History(qb.Securities.Keys, qb.Time-TimeSpan.FromDays(10), qb.Time, dataNormalizationMode: DataNormalizationMode.SplitAdjusted);
history = qb.History(qb.Securities.Keys, qb.Time-timedelta(days=10), qb.Time, dataNormalizationMode=DataNormalizationMode.SplitAdjusted)

Wrangle Data

You need some historical data to perform wrangling operations. The process to manipulate the historical data depends on its data type. To display pandas objects, run a cell in a notebook with the pandas object as the last line. To display other data formats, call the print method.

You need some historical data to perform wrangling operations. Use LINQ to wrangle the data and then call the Console.WriteLine method in a Jupyter Notebook to display the data. The process to manipulate the historical data depends on its data type.

DataFrame Objects

If the History method returns a DataFrame, the first level of the DataFrame index is the Equity Symbol and the second level is the EndTime of the data sample. The columns of the DataFrame are the data properties.

DataFrame of two Equities

To select the historical data of a single Equity, index the loc property of the DataFrame with the Equity Symbol.

all_history_df.loc[spy]  # or all_history_df.loc['SPY']
DataFrame of one Equity

To select a column of the DataFrame, index it with the column name.

all_history_df.loc[spy]['close']
Series of close values

If you request historical data for multiple Equities, you can transform the DataFrame so that it's a time series of close values for all of the Equities. To transform the DataFrame, select the column you want to display for each Equity and then call the unstack method.

all_history_df['close'].unstack(level=0)

The DataFrame is transformed so that the column indices are the Symbol of each Equity and each row contains the close value.

If you prefer to display the ticker of each Symbol instead of the string representation of the SecurityIdentifier, follow these steps:

  1. Create a dictionary where the keys are the string representations of each SecurityIdentifier and the values are the ticker.
  2. tickers_by_id = {str(x.ID): x.Value for x in qb.Securities.Keys}
  3. Get the values of the symbol level of the DataFrame index and create a list of tickers.
  4. tickers = set([tickers_by_id[x] for x in all_history_df.index.get_level_values('symbol')])
  5. Set the values of the symbol level of the DataFrame index to the list of tickers.
  6. all_history_df.index.set_levels(tickers, 'symbol', inplace=True)

The new DataFrame is keyed by the ticker.

all_history_df.loc[spy.Value]  # or all_history_df.loc["SPY"]  

After the index renaming, the unstacked DataFrame has the following format:

Slice Objects

If the History method returns Slice objects, iterate through the Slice objects to get each one. The Slice objects may not have data for all of your Equity subscriptions. To avoid issues, check if the Slice contains data for your Equity before you index it with the Equity Symbol.

foreach (var slice in allHistorySlice)
{
    if (slice.Bars.ContainsKey(spy))
    {
        var tradeBar = slice.Bars[spy];
    }
    if (slice.QuoteBars.ContainsKey(spy))
    {
        var quoteBar = slice.QuoteBars[spy];
    }
}
for slice in all_history_slice:
    if slice.Bars.ContainsKey(spy):
        trade_bar = slice.Bars[spy]
    if slice.QuoteBars.ContainsKey(spy):
        quote_bar = slice.QuoteBars[spy]

You can also iterate through each TradeBar and QuoteBar in the Slice.

foreach (var slice in allHistorySlice)
{
    foreach (var kvp in slice.Bars)
    {
        var symbol = kvp.Key;
        var tradeBar = kvp.Value;
    }
    foreach (var kvp in slice.QuoteBars)
    {
        var symbol = kvp.Key;
        var quoteBar = kvp.Value;
    }
}
for slice in all_history_slice:
    for kvp in slice.Bars:
        symbol = kvp.Key
        trade_bar = kvp.Value
    for kvp in slice.QuoteBars:
        symbol = kvp.Key
        quote_bar = kvp.Value

You can also use LINQ to select each TradeBar in the Slice for a given Symbol

var tradeBars = allHistorySlice.Where(slice => slice.Bars.ContainsKey(spy)).Select(slice => slice.Bars[spy]);

TradeBar Objects

If the History method returns TradeBar objects, iterate through the TradeBar objects to get each one.

foreach (var tradeBar in singleHistoryTradeBars)
{
    Console.WriteLine(tradeBar);
}
for trade_bar in single_history_trade_bars:
    print(trade_bar)

If the History method returns TradeBars, iterate through the TradeBars to get the TradeBar of each Equity. The TradeBars may not have data for all of your Equity subscriptions. To avoid issues, check if the TradeBars object contains data for your security before you index it with the Equity Symbol.

foreach (var tradeBars in allHistoryTradeBars)
{
    if (tradeBars.ContainsKey(spy))
    {
        var tradeBar = tradeBars[spy];
    }
}
for trade_bars in all_history_trade_bars:
    if trade_bars.ContainsKey(spy):
        trade_bar = trade_bars[spy]

You can also iterate through each of the TradeBars.

foreach (var tradeBars in allHistoryTradeBars)
{
    foreach (var kvp in tradeBars)
    {
        var symbol = kvp.Key;
        var tradeBar = kvp.Value;
    }
}
for trade_bars in all_history_trade_bars:
    for kvp in trade_bars:
        symbol = kvp.Key
        trade_bar = kvp.Value

QuoteBar Objects

If the History method returns QuoteBar objects, iterate through the QuoteBar objects to get each one.

foreach (var quoteBar in singleHistoryQuoteBars)
{
    Console.WriteLine(quoteBar);
}
for quote_bar in single_history_quote_bars:
    print(quote_bar)

If the History method returns QuoteBars, iterate through the QuoteBars to get the QuoteBar of each Equity. The QuoteBars may not have data for all of your Equity subscriptions. To avoid issues, check if the QuoteBars object contains data for your security before you index it with the Equity Symbol.

foreach (var quoteBars in allHistoryQuoteBars)
{
    if (quoteBars.ContainsKey(spy))
    {
        var quoteBar = quoteBars[spy];
    }
}
for quote_bars in all_history_quote_bars:
    if quote_bars.ContainsKey(spy):
        quote_bar = quote_bars[spy]

You can also iterate through each of the QuoteBars.

foreach (var quoteBars in allHistoryQuoteBars)
{
    foreach (var kvp in quoteBars)
    {
        var symbol = kvp.Key;
        var quoteBar = kvp.Value;
    }
}
for quote_bars in all_history_quote_bars:
    for kvp in quote_bars:
        symbol = kvp.Key
        quote_bar = kvp.Value

Tick Objects

If the History method returns Tick objects, iterate through the Tick objects to get each one.

foreach (var tick in singleHistoryTicks)
{
    Console.WriteLine(tick);
}
for tick in single_history_ticks:
    print(tick)

If the History method returns Ticks, iterate through the Ticks to get the Tick of each Equity. The Ticks may not have data for all of your Equity subscriptions. To avoid issues, check if the Ticks object contains data for your security before you index it with the Equity Symbol.

foreach (var ticks in allHistoryTicks)
{
    if (ticks.ContainsKey(spy))
    {
        var tick = ticks[spy];
    }
}
for ticks in all_history_ticks:
    if ticks.ContainsKey(spy):
        ticks = ticks[spy]

You can also iterate through each of the Ticks.

foreach (var ticks in allHistoryTicks)
{
    foreach (var kvp in ticks)
    {
        var symbol = kvp.Key;
        var tick = kvp.Value;
    }
}
for ticks in all_history_ticks:
    for kvp in ticks:
        symbol = kvp.Key
        tick = kvp.Value

Plot Data

Jupyter Notebooks don't currently support libraries to plot historical data, but we are working on adding the functionality. Until we add the functionality, use Python to plot historical Equity data.

You need some historical Equity data to produce plots. You can use many of the supported plotting libraries to visualize data in various formats. For example, you can plot candlestick and line charts.

Candlestick Chart

Follow these steps to plot candlestick charts:

  1. Get some historical data.
  2. history = qb.History(spy, datetime(2021, 11, 23), datetime(2021, 12, 8), Resolution.Daily).loc[spy]
  3. Import the plotly library.
  4. import plotly.graph_objects as go
  5. Create a Candlestick.
  6. candlestick = go.Candlestick(x=history.index,
                                 open=history['open'],
                                 high=history['high'],
                                 low=history['low'],
                                 close=history['close'])
  7. Create a Layout.
  8. layout = go.Layout(title=go.layout.Title(text='SPY OHLC'),
                       xaxis_title='Date',
                       yaxis_title='Price',
                       xaxis_rangeslider_visible=False)
  9. Create the Figure.
  10. fig = go.Figure(data=[candlestick], layout=layout)
  11. Show the Figure.
  12. fig.show()

    Candlestick charts display the open, high, low, and close prices of the security.

Line Chart

Follow these steps to plot line charts using built-in methods:

  1. Get some historical data.
  2. history = qb.History([spy, tlt], datetime(2021, 11, 23), datetime(2021, 12, 8), Resolution.Daily)
  3. Select the data to plot.
  4. volume = history['volume'].unstack(level=0)
  5. Call the plot method on the pandas object.
  6. volume.plot(title="Volume", figsize=(15, 10))
  7. Show the plot.
  8. plt.show()

    Line charts display the value of the property you selected in a time series.

Common Errors

Some factor files have INF split values, which indicate that the stock has so many splits that prices can't be calculated with correct numerical precision. To allow history requests with these symbols, we need to move the starting date forward when reading the data. If there are numerical precision errors in the factor files for a security in your history request, LEAN throws the following error:

"Warning: when performing history requests, the start date will be adjusted if there are numerical precision errors in the factor files."

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: