Datasets

Indices

Introduction

This page explains how to request, manipulate, and visualize historical Index data.

Create Subscriptions

Follow these steps to subscribe to an Index security:

  1. Load the assembly files and data types in their own cell.
    2. #load "../Initialize.csx"
  2. Import the data types.
    3. #load "../QuantConnect.csx"
#r "../Microsoft.Data.Analysis.dll"

using QuantConnect;
using QuantConnect.Data;
using QuantConnect.Algorithm;
using QuantConnect.Research;
using QuantConnect.Indicators;
using QuantConnect.Securities.Index;
using Microsoft.Data.Analysis;
  3. Create a QuantBook.
    4. var qb = new QuantBook();
    qb = QuantBook()
  4. Call the AddIndexadd_index method with a ticker and then save a reference to the Index Symbol.
    5. var spx = qb.AddIndex("SPX").Symbol;
var vix = qb.AddIndex("VIX").Symbol;
    spx = qb.add_index("SPX").symbol
vix = qb.add_index("VIX").symbol

To view all of the available indices, see Supported Indices.

Get Historical Data

You need a subscription before you can request historical data for a security. On the time dimension, you can request an amount of historical data based on a trailing number of bars, a trailing period of time, or a defined period of time. On the security dimension, you can request historical data for a single Index, a subset of the Indices you created subscriptions for in your notebook, or all of the Indices in your notebook.

Trailing Number of Bars

To get historical data for a number of trailing bars, call the Historyhistory method with the Symbol object(s) and an integer.

// Slice objects
var singleHistorySlice = qb.History(spx, 10);
var subsetHistorySlice = qb.History(new[] {spx, vix}, 10);
var allHistorySlice = qb.History(10);

// TradeBar objects
var singleHistoryTradeBars = qb.History<TradeBar>(spx, 10);
var subsetHistoryTradeBars = qb.History<TradeBar>(new[] {spx, vix}, 10);
var allHistoryTradeBars = qb.History<TradeBar>(qb.Securities.Keys, 10);
# DataFrame
single_history_df = qb.history(spx, 10)
single_history_trade_bar_df = qb.history(TradeBar, spx, 10)
subset_history_df = qb.history([spx, vix], 10)
subset_history_trade_bar_df = qb.history(TradeBar, [spx, vix], 10)
all_history_df = qb.history(qb.securities.keys(), 10)
all_history_trade_bar_df = qb.history(TradeBar, qb.securities.keys(), 10)

# Slice objects
all_history_slice = qb.history(10)

# TradeBar objects
single_history_trade_bars = qb.history[TradeBar](spx, 10)
subset_history_trade_bars = qb.history[TradeBar]([spx, vix], 10)
all_history_trade_bars = qb.history[TradeBar](qb.securities.keys(), 10)

The preceding calls return the most recent bars, excluding periods of time when the exchange was closed.

Trailing Period of Time

To get historical data for a trailing period of time, call the Historyhistory method with the Symbol object(s) and a TimeSpantimedelta.

// Slice objects
var singleHistorySlice = qb.History(spx, TimeSpan.FromDays(3));
var subsetHistorySlice = qb.History(new[] {spx, vix}, TimeSpan.FromDays(3));
var allHistorySlice = qb.History(10);

// TradeBar objects
var singleHistoryTradeBars = qb.History<TradeBar>(spx, TimeSpan.FromDays(3));
var subsetHistoryTradeBars = qb.History<TradeBar>(new[] {spx, vix}, TimeSpan.FromDays(3));
var allHistoryTradeBars = qb.History<TradeBar>(TimeSpan.FromDays(3));

// Tick objects
var singleHistoryTicks = qb.History<Tick>(spx, TimeSpan.FromDays(3), Resolution.Tick);
var subsetHistoryTicks = qb.History<Tick>(new[] {spx, vix}, TimeSpan.FromDays(3), Resolution.Tick);
var allHistoryTicks = qb.History<Tick>(qb.Securities.Keys, TimeSpan.FromDays(3), Resolution.Tick);
# DataFrame of trade data (indices don't have quote data)
single_history_df = qb.history(spx, timedelta(days=3))
subset_history_df = qb.history([spx, vix], timedelta(days=3))
all_history_df = qb.history(qb.securities.keys(), timedelta(days=3))

# DataFrame of tick data
single_history_tick_df = qb.history(spx, timedelta(days=3), Resolution.TICK)
subset_history_tick_df = qb.history([spx, usb], timedelta(days=3), Resolution.TICK)
all_history_tick_df = qb.history(qb.securities.keys(), timedelta(days=3), Resolution.TICK)

# Slice objects
all_history_slice = qb.history(timedelta(days=3))

# TradeBar objects
single_history_trade_bars = qb.history[TradeBar](spx, timedelta(days=3))
subset_history_trade_bars = qb.history[TradeBar]([spx, vix], timedelta(days=3))
all_history_trade_bars = qb.history[TradeBar](qb.securities.keys(), timedelta(days=3))

# Tick objects
single_history_ticks = qb.history[Tick](spx, timedelta(days=3), Resolution.TICK)
subset_history_ticks = qb.history[Tick]([spx, vix], timedelta(days=3), Resolution.TICK)
all_history_ticks = qb.history[Tick](qb.securities.keys(), timedelta(days=3), Resolution.TICK)

The preceding calls return the most recent bars or ticks, excluding periods of time when the exchange was closed.

Defined Period of Time

To get historical data for a specific period of time, call the Historyhistory method with the Symbol object(s), a start DateTimedatetime, and an end DateTimedatetime. The start and end times you provide are based in the notebook time zone.

var startTime = new DateTime(2021, 1, 1);
var endTime = new DateTime(2021, 2, 1);

// Slice objects
var singleHistorySlice = qb.History(spx, startTime, endTime);
var subsetHistorySlice = qb.History(new[] {spx, vix}, startTime, endTime);
var allHistorySlice = qb.History(qb.Securities.Keys, startTime, endTime);

// TradeBar objects
var singleHistoryTradeBars = qb.History<TradeBar>(spx, startTime, endTime);
var subsetHistoryTradeBars = qb.History<TradeBar>(new[] {spx, vix}, startTime, endTime);
var allHistoryTradeBars = qb.History<TradeBar>(qb.Securities.Keys, startTime, endTime);

// Tick objects
var singleHistoryTicks = qb.History<Tick>(spx, startTime, endTime, Resolution.Tick);
var subsetHistoryTicks = qb.History<Tick>(new[] {spx, vix}, startTime, endTime, Resolution.Tick);
var allHistoryTicks = qb.History<Tick>(qb.Securities.Keys, startTime, endTime, Resolution.Tick);
start_time = datetime(2021, 1, 1)
end_time = datetime(2021, 2, 1)

# DataFrame of trade data (indices don't have quote data)
single_history_df = qb.history(spx, start_time, end_time)
subset_history_df = qb.history([spx, vix], start_time, end_time)
all_history_df = qb.history(qb.securities.keys(), start_time, end_time)

# DataFrame of tick data
single_history_tick_df = qb.history(spx, start_time, end_time, Resolution.TICK)
subset_history_tick_df = qb.history([spx, vix], start_time, end_time, Resolution.TICK)
all_history_tick_df = qb.history(qb.securities.keys(), start_time, end_time, Resolution.TICK)

# TradeBar objects
single_history_trade_bars = qb.history[TradeBar](spx, start_time, end_time)
subset_history_trade_bars = qb.history[TradeBar]([spx, vix], start_time, end_time)
all_history_trade_bars = qb.history[TradeBar](qb.securities.keys(), start_time, end_time)

# Tick objects
single_history_ticks = qb.history[Tick](spx, start_time, end_time, Resolution.TICK)
subset_history_ticks = qb.history[Tick]([spx, vix], start_time, end_time, Resolution.TICK)
all_history_ticks = qb.history[Tick](qb.securities.keys(), start_time, end_time, Resolution.TICK)

The preceding calls return the bars or ticks that have a timestamp within the defined period of time.

Resolutions

The following table shows the available resolutions and data formats for Index subscriptions:

ResolutionTradeBarQuoteBarTrade TickQuote Tick
TickTICK
SecondSECOND
MinuteMINUTEgreen check
HourHOURgreen check
DailyDAILYgreen check

Markets

The only market available for Indices is Market.USA.

Data Normalization

The data normalization mode doesn't affect data from history request. If you change the data normalization mode, it won't change the outcome.

Wrangle Data

You need some historical data to perform wrangling operations. The process to manipulate the historical data depends on its data type. To display pandas objects, run a cell in a notebook with the pandas object as the last line. To display other data formats, call the print method.

You need some historical data to perform wrangling operations. Use LINQ to wrangle the data and then call the Console.WriteLine method in a Jupyter Notebook to display the data. The process to manipulate the historical data depends on its data type.

DataFrame Objects

If the Historyhistory method returns a DataFrame, the first level of the DataFrame index is the encoded Index Symbol and the second level is the EndTimeend_time of the data sample. The columns of the DataFrame are the data properties.

DataFrame of two Indices

To select the historical data of a single Index, index the loc property of the DataFrame with the Index Symbol.

all_history_df.loc[spx]  # or all_history_df.loc['SPX']
DataFrame of one Index

To select a column of the DataFrame, index it with the column name.

all_history_df.loc[spx]['close']
Series of close values

If you request historical data for multiple Indices, you can transform the DataFrame so that it's a time series of close values for all of the Indices. To transform the DataFrame, select the column you want to display for each Index and then call the unstack method.

all_history_df['close'].unstack(level=0)

The DataFrame is transformed so that the column indices are the Symbol of each Index and each row contains the close value.

DataFrame of one Index

The historical data methods don't return DataFrame objects, but you can create one for efficient vectorized data wrangling.

using Microsoft.Data.Analysis; 

var columns = new DataFrameColumn[] {
    new PrimitiveDataFrameColumn("Time", history.Select(x => x[spx].EndTime)),
    new DecimalDataFrameColumn("SPX Open", history.Select(x => x[spx].Open)),
    new DecimalDataFrameColumn("SPX High", history.Select(x => x[spx].High)),
    new DecimalDataFrameColumn("SPX Low", history.Select(x => x[spx].Low)),
    new DecimalDataFrameColumn("SPX Close", history.Select(x => x[spx].Close))
};
var df = new DataFrame(columns);
df
Historical C# dataframe

To select a particular column of the DataFrame, index it with the column name.

df["SPX close"]
Historical C# dataframe column

Slice Objects

If the Historyhistory method returns Slice objects, iterate through the Slice objects to get each one. The Slice objects may not have data for all of your Index subscriptions. To avoid issues, check if the Slice contains data for your Index before you index it with the Index Symbol.

foreach (var slice in allHistorySlice) {
    if (slice.Bars.ContainsKey(spx))
    {
        var tradeBar = slice.Bars[spx];
    }
}
for slice in all_history_slice:
        if slice.bars.contains_key(spx):
        trade_bar = slice.bars[spx]

You can also iterate through each TradeBar in the Slice.

foreach (var slice in allHistorySlice)
{
    foreach (var kvp in slice.Bars)
    {
        var symbol = kvp.Key;
        var tradeBar = kvp.Value;
    }
}
for slice in all_history_slice:
    for kvp in slice.bars:
        symbol = kvp.key
        trade_bar = kvp.value

You can also use LINQ to select each TradeBar in the Slice for a given Symbol.

var tradeBars = allHistorySlice.Where(slice => slice.Bars.ContainsKey(spx)).Select(slice => slice.Bars[spx]);

TradeBar Objects

If the Historyhistory method returns TradeBar objects, iterate through the TradeBar objects to get each one.

foreach (var tradeBar in singleHistoryTradeBars)
{
    Console.WriteLine(tradeBar);
}
for trade_bar in single_history_trade_bars:
    print(trade_bar)

If the Historyhistory method returns TradeBars, iterate through the TradeBars to get the TradeBar of each Index. The TradeBars may not have data for all of your Index subscriptions. To avoid issues, check if the TradeBars object contains data for your security before you index it with the Index Symbol.

foreach (var tradeBars in allHistoryTradeBars)
{
    if (tradeBars.ContainsKey(spx))
    {
        var tradeBar = tradeBars[spx];
    }
}
for trade_bars in all_history_trade_bars:
    if trade_bars.contains_key(spx):
        trade_bar = trade_bars[spx]

You can also iterate through each of the TradeBars.

foreach (var tradeBars in allHistoryTradeBars)
{
    foreach (var kvp in tradeBars)
    {
        var symbol = kvp.Key;
        var tradeBar = kvp.Value;
    }
}
for trade_bars in all_history_trade_bars:
    for kvp in trade_bars:
        symbol = kvp.Key
        trade_bar = kvp.Value

Tick Objects

If the Historyhistory method returns TickTICK objects, iterate through the TickTICK objects to get each one.

foreach (var tick in singleHistoryTicks)
{
    Console.WriteLine(tick);
}
for tick in single_history_ticks:
    print(tick)

If the Historyhistory method returns Ticks, iterate through the Ticks to get the TickTICK of each Index. The Ticks may not have data for all of your Index subscriptions. To avoid issues, check if the Ticks object contains data for your security before you index it with the Index Symbol.

foreach (var ticks in allHistoryTicks)
{
    if (ticks.ContainsKey(spx))
    {
        var tick = ticks[spx];
    }
}
for ticks in all_history_ticks:
    if ticks.contains_key(spx):
        ticks = ticks[spx]

You can also iterate through each of the Ticks.

foreach (var ticks in allHistoryTicks)
{
    foreach (var kvp in ticks)
    {
        var symbol = kvp.Key;
        var tick = kvp.Value;
    }
}
for ticks in all_history_ticks:
    for kvp in ticks:
        symbol = kvp.key
        tick = kvp.value

The Ticks objects only contain the last tick of each security for that particular timeslice

Plot Data

You need some historical Indices data to produce plots. You can use many of the supported plotting librariesPlot.NET package to visualize data in various formats. For example, you can plot candlestick and line charts.

Candlestick Chart

Follow these steps to plot candlestick charts:

  1. Get some historical data.
    2. history = qb.history(spx, datetime(2021, 11, 24), datetime(2021, 12, 8), Resolution.DAILY).loc[spx]
    var history = qb.History<TradeBar>(spx, new DateTime(2021, 11, 24), new DateTime(2021, 12, 8), Resolution.Daily);
  2. Import the plotlyPlot.NET library.
    3. import plotly.graph_objects as go
    #r "../Plotly.NET.dll"
using Plotly.NET;
using Plotly.NET.LayoutObjects;
  3. Create a Candlestick.
    4. candlestick = go.Candlestick(x=history.index,
                             open=history['open'],
                             high=history['high'],
                             low=history['low'],
                             close=history['close'])
    var chart = Chart2D.Chart.Candlestick<decimal, decimal, decimal, decimal, DateTime, string>(
    history.Select(x => x.Open),
    history.Select(x => x.High),
    history.Select(x => x.Low),
    history.Select(x => x.Close),
    history.Select(x => x.EndTime)
);
  4. Create a Layout.
    5. layout = go.Layout(title=go.layout.Title(text='SPX OHLC'),
                   xaxis_title='Date',
                   yaxis_title='Price',
                   xaxis_rangeslider_visible=False)
    LinearAxis xAxis = new LinearAxis();
xAxis.SetValue("title", "Time");
LinearAxis yAxis = new LinearAxis();
yAxis.SetValue("title", "Price ($)");
Title title = Title.init($"{spx} OHLC");

Layout layout = new Layout();
layout.SetValue("xaxis", xAxis);
layout.SetValue("yaxis", yAxis);
layout.SetValue("title", title);
  5. Create a Figure.
    6. fig = go.Figure(data=[candlestick], layout=layout)
  6. Assign the Layout to the chart.
    7. chart.WithLayout(layout);
  7. Show the Figure.
    8. fig.show()
    HTML(GenericChart.toChartHTML(chart))

    Candlestick charts display the open, high, low, and close prices of the security.

Candlestick plot of SPX OHLC Candlestick plot of SPX OHLC

Line Chart

Follow these steps to plot line charts using built-in methodsPlotly.NET package:

  1. Get some historical data.
    2. history = qb.history([spx, vix], datetime(2021, 11, 24), datetime(2021, 12, 8), Resolution.DAILY)
    var history = qb.History<TradeBar>(new [] {spx, vix}, new DateTime(2021, 11, 24), new DateTime(2021, 12, 8), Resolution.Daily);
  2. Select the data to plot.
    3. pct_change = history['close'].unstack(0).pct_change().dropna()
  3. Call the plot method on the pandas object.
    4. pct_change.plot(title="Close Price %Change", figsize=(15, 10))
  4. Create Line charts.
    5. var chart1 = Chart2D.Chart.Line<DateTime, decimal, string>(
    history.Select(x => x[spx].EndTime),
    history.Select(x => x[spx].Close - x[spx].Open),
    Name: $"{spx}"
);
var chart2 = Chart2D.Chart.Line<DateTime, decimal, string>(
    history.Select(x => x[vix].EndTime),
    history.Select(x => x[vix].Close - x[vix].Open),
    Name: $"{vix}"
);
  5. Create a Layout.
    6. LinearAxis xAxis = new LinearAxis();
xAxis.SetValue("title", "Time");
LinearAxis yAxis = new LinearAxis();
yAxis.SetValue("title", "Price ($)");
Title title = Title.init($"{spx} & {vix} Daily Spread");

Layout layout = new Layout();
layout.SetValue("xaxis", xAxis);
layout.SetValue("yaxis", yAxis);
layout.SetValue("title", title);
  6. Combine the charts and assign the Layout to the chart.
    7. var chart = Plotly.NET.Chart.Combine(new []{chart1, chart2});
chart.WithLayout(layout);
  7. Show the plot.
    8. plt.show()
    HTML(GenericChart.toChartHTML(chart))

    Line charts display the value of the property you selected in a time series.

Line chart of indices %chg Line chart of indices daily spread

Examples

The following examples demonstrate some common practices for applying the US Indices dataset.

Example 1: 5-Minute Candlestick Plot

The following example studies the candlestick pattern of the SPX. To study the short term pattern, we consolidate the data into 5 minute bars and plot the 5-minute candlestick plot.

// Load the required assembly files and data types in a separate cell.
#load "../Initialize.csx"

#load "../QuantConnect.csx"
using System;
using QuantConnect;
using QuantConnect.Data;
using QuantConnect.Data.Market;
using QuantConnect.Data.Consolidators;
using QuantConnect.Indicators;
using QuantConnect.Algorithm;
using QuantConnect.Research;
using QuantConnect.Securities;

// Import Plotly for plotting.
#r "../Plotly.NET.dll"
using Plotly.NET;
using Plotly.NET.LayoutObjects;

// Create a QuantBook.
var qb = new QuantBook();

// Request the SPX historical data.
var symbol = qb.AddIndex("SPX").Symbol;
var history = qb.History<TradeBar>(symbol, 
    start: qb.Time - TimeSpan.FromDays(182),
    end: qb.Time,
    resolution: Resolution.Minute);

// Set up a consolidator and a RollingWindow to save the data
var consolidator = new TradeBarConsolidator(TimeSpan.FromMinutes(5));
var window = new RollingWindow<TradeBar>(10000);

// Attach a consolidation handler method that saves the consolidated bars in the RollingWindow
consolidator.DataConsolidated += (sender, consolidated) =>
{
    window.Add(consolidated);
};

// Iterate the historical market data and feed each bar into the consolidator
foreach(var bar in history)
{
    consolidator.Update(bar);
}

// Crete the Candlestick chart using the 5-minute windows.
var chart = Chart2D.Chart.Candlestick<decimal, decimal, decimal, decimal, DateTime, string>(
    window.Select(x => x.Open),
    window.Select(x => x.High),
    window.Select(x => x.Low),
    window.Select(x => x.Close),
    window.Select(x => x.EndTime)
);
// Create a Layout as the plot settings.
LinearAxis xAxis = new LinearAxis();
xAxis.SetValue("title", "Time");
LinearAxis yAxis = new LinearAxis();
yAxis.SetValue("title", "Price ($)");
Title title = Title.init($"{symbol} Price");

Layout layout = new Layout();
layout.SetValue("xaxis", xAxis);
layout.SetValue("yaxis", yAxis);
layout.SetValue("title", title);
// Assign the Layout to the chart.
chart.WithLayout(layout);
// Display the plot.
HTML(GenericChart.toChartHTML(chart))
# Import plotly library for plotting.
import plotly.graph_objects as go

# Create a QuantBook
qb = QuantBook()

# Request SPX's historical data.
symbol = qb.add_index("SPX").symbol
history = qb.history(symbol, 
    start=qb.time - timedelta(days=182), 
    end=qb.time, 
    resolution=Resolution.MINUTE)

# Drop level 0 index (Symbol index) from the DataFrame
history = history.droplevel([0])

# Select the required columns to obtain the 5-minute OHLC data.
history = history[["open", "high", "low", "close"]].resample("5T").agg({
    "open": "first",
    "high": "max",
    "low": "min",
    "close": "last"
})

# Crete the Candlestick chart using the 5-minute windows.
candlestick = go.Candlestick(x=history.index,
                             open=history['open'],
                             high=history['high'],
                             low=history['low'],
                             close=history['close'])
# Create a Layout as the plot settings.
layout = go.Layout(title=go.layout.Title(text=f'{symbol} OHLC'),
                   xaxis_title='Date',
                   yaxis_title='Price',
                   xaxis_rangeslider_visible=False)
# Create the Figure.
fig = go.Figure(data=[candlestick], layout=layout)
# Display the plot.
fig.show()

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: