Datasets

Working knowledge of C#.

Working knowedge of Python and pandas. If you are not familiar with pandas, see the pandas documentation.

Follow these steps to subscribe to a Future security:

1. Load the required assembly files and data types.

#load "../Initialize.csx"
#load "../QuantConnect.csx"

using QuantConnect;
using QuantConnect.Data;
using QuantConnect.Data.Market;
using QuantConnect.Algorithm;
using QuantConnect.Securities;
using QuantConnect.Securities.Future;
using QuantConnect.Research;

2. Instantiate a QuantBook.

var qb = new QuantBook();

qb = QuantBook()

3. Call the AddFuture method with a ticker, resolution, and contract rollover settings.

var future = qb.AddFuture(Futures.Indices.SP500EMini, Resolution.Minute,
                dataNormalizationMode: DataNormalizationMode.BackwardsRatio,
                dataMappingMode: DataMappingMode.LastTradingDay,
                contractDepthOffset: 0
            );

future = qb.AddFuture(Futures.Indices.SP500EMini, Resolution.Minute,
                dataNormalizationMode = DataNormalizationMode.BackwardsRatio,
                dataMappingMode = DataMappingMode.LastTradingDay,
                contractDepthOffset = 0)

If you do not pass a resolution argument, Resolution.Minute is used by default. If you do not pass rollover setting arguments, the default values of dataNormalizationMode, dataMappingMode, and contractDepthOffset are DataNormalizationMode.Adjusted, DataMappingMode.OpenInterest, and 0, respectively.

4. Save a reference to the Future Symbol.

var symbol = future.Symbol;

symbol = future.Symbol

The Future Symbol references the continuous contract.

5. (Optional) Call the SetFilter method with a specified expiration range value.

future.SetFilter(0, 90);

future.SetFilter(0, 90)

If you don't call the SetFilter method, the GetFutureHistory method will not return historical data.

If you want historical data on individual contracts and their OpenInterest, you need to subscribe to the individual contracts. You need to subscribe to the underlying Future before you can subscribe to its contracts. Follow these steps to subscribe to individual Future contracts:

1. Call the GetFuturesContractList method with the underlying Future Symbol and a datetimeDateTime.

var startDate = new DateTime(2021,12,20);
var symbols = qb.FutureChainProvider.GetFutureContractList(symbol, startDate);

start_date = datetime(2021,12,20)
symbols = qb.FutureChainProvider.GetFutureContractList(symbol, start_date)

The call returns a list of Symbols that reference the FutureContracts that were trading at the given time.

2. Select the Symbol of the FutureContract(s) for which you want to get historical data.

For example, select the Symbol of the contract with the closest expiry to match the continuous Future defined above.


var contractSymbol = symbols.OrderBy(s => s.ID.Date).FirstOrDefault();

contract_symbol = sorted(symbols, key=lambda s: s.ID.Date)[0]

3. Call the AddFutureContract method with an FutureContract Symbol and disable fill-forward.

qb.AddFutureContract(contractSymbol, fillDataForward: false);

qb.AddFutureContract(contract_symbol, fillDataForward = False)

Disable fill-forward because there are only a few OpenInterest data points per day.

You need a subscription before you can request historical data for a security. You can request historical data for the continuous Future or Future contracts. You can request an amount of historical data based on a trailing number of bars, a trailing period of time, or a defined period of time. To reduce the risk of look-ahead bias, before you request historical data, call the SetStartDate method with a datetimeDateTime.

qb.SetStartDate(startDate);

qb.SetStartDate(start_date)

The datetimeDateTime passed to the SetStartDate method is the most recent day for which you can receive historical data. You can not receive historical data that is timestamped after the datetimeDateTime.

Defined Period of Time

Call the History method with a symbol, start DateTimedatetime, and end DateTimedatetime to request historical data based on the defined period of time.

var startTime = new DateTime(2021,12,1);
var endTime = new DateTime(2021,12,31);
var continuousHistory = qb.History(symbol, startTime, endTime);
var contractHistory = qb.History(contractSymbol, startTime, endTime);
var openInterestHistory = qb.History<OpenInterest>(contractSymbol, startTime, endTime);

start_time = datetime(2021,12,1)
end_time = datetime(2021,12,31)
continuous_history = qb.History(symbol, start_time, end_time)
contract_history = qb.History(contract_symbol, start_time, end_time)
open_interest_history = qb.History(OpenInterest, contract_symbol, start_time, end_time)

These calls return the bars that are timestamped within the defined period of time.

Trailing Number of Bars

Call the History method with a symbol and integer to request historical data based on the given number of trailing bars.

var continuousHistory = qb.History(symbol, 400);
var contractHistory = qb.History(contractSymbol, 400);
var openInterestHistory = qb.History<OpenInterest>(contractSymbol, 400);
	

continuous_history = qb.History(symbol, 400)
contract_history = qb.History(contract_symbol, 400)
open_interest_history = qb.History(OpenInterest, contract_symbol, 400)

These call return the most recent bars, excluding periods of time when the exchange was closed. If you called the StartDate method, these calls may take longer to return data.

Trailing Period of Time

Call the History method with a symbol and TimeSpantimedelta to request historical data based on the given trailing period of time.

var continuousHistory = qb.History(symbol, TimeSpan.FromDays(10));
var contractHistory = qb.History(contractSymbol, TimeSpan.FromDays(10));
var openInterestHistory = qb.History<OpenInterest>(contractSymbol, TimeSpan.FromDays(10));

continuous_history = qb.History(symbol, timedelta(days=10))
contract_history = qb.History(contract_symbol, timedelta(days=10))
open_interest_history = qb.History(OpenInterest, contract_symbol, timedelta(days=10))

These calls return the most recent bars, excluding periods of time when the exchange was closed.

In all of the cases above, the History method returns a DataFrame with a MultiIndex. If you request history for the continuous Future, the DataFrame contains quote prices, trade prices, and volume data.

If you request history for a Futures contract, the DataFrame contains quote prices, trade prices, and volume data.

If you request history for the open interest of a Futures contract, the DataFrame contains open interest data.

In all of the cases above, the History method returns an IEnumerable<TradeBar> for the market price history requests and an IEnumerable<OpenInterest> for open interest history request.

The History method returns an IEnumerable<Slice> for the market price history requests, if the first argument is a list. For example, the defined period of time case will be coded as follows:

var history = qb.History(new [] { symbol }, startTime, endTime);
var contractHistory = qb.History(new [] { contractSymbol }, startTime, endTime);

You need to subscribe to all of the Futures for which you want historical data. You can request historical data for a subset of the Option contract properties or a subset of the Option contracts.

Subset of Futures

Create subscriptions for multiple Futures and then call the History method with a list of Symbols to request historical data for those specific securities.

1. Call the AddFuture method with a ticker, resolution, and contract rollover settings.

var es = qb.AddFuture(Futures.Indices.SP500EMini).Symbol;
var nq = qb.AddFuture(Futures.Indices.NASDAQ100EMini).Symbol;
var ym = qb.AddFuture(Futures.Indices.Dow30EMini).Symbol;

es = qb.AddFuture(Futures.Indices.SP500EMini).Symbol
nq = qb.AddFuture(Futures.Indices.NASDAQ100EMini).Symbol
ym = qb.AddFuture(Futures.Indices.Dow30EMini).Symbol

If you do not pass a resolution argument, Resolution.Minute is used by default. If you do not pass rollover setting arguments, the default values of dataNormalizationMode, dataMappingMode, and contractDepthOffset are DataNormalizationMode.Adjusted, DataMappingMode.OpenInterest, and 0, respectively.

2. Call the History method with a list of Symbols to get the historical data of the given Symbols.

var definedPeriodContinuousHistory = qb.History(new []{es, nq, ym}, startTime, endTime, Resolution.Minute);
var timespanContinuousHistory = qb.History(new []{es, nq, ym}, TimeSpan.FromDays(10), Resolution.Minute);
var barCountContinuousHistory = qb.History(new []{es, nq, ym}, 400, Resolution.Minute);

defined_period_continuous_history = qb.History([es, nq, ym], start_time, end_time, Resolution.Minute)
timespan_continuous_history = qb.History([es, nq, ym], timedelta(days=10), Resolution.Minute)
bar_count_continuous_history = qb.History([es, nq, ym], 400, Resolution.Minute)

Subset of Futures Contract Properties

This functionality is not currently supported for C# research environments. Use Python to get a subset of FutureContract properties.

Call the GetFutureHistory method with a Future Symbol, start datetime, end datetime, and resolution to get historical Future contract data for the given Symbol.

future_history = qb.GetFutureHistory(symbol, start_time, end_time, Resolution.Minute)

The call returns an FutureHistory object. If you don't call the SetFilter method, the GetFutureHistory method will not return data since GetFutureHistory uses the filter definition to select contracts and their historical data.

Call the GetAllData method to get all of the historical data for the selected contracts and the underlying Equity.

history = future_history.GetAllData()

The call returns a DataFrame with a MultiIndex.

Call the GetExpiryDates method to get all of the expiry dates for the selected contracts.

expiries = future_history.GetExpiryDates()

The call returns a list of datetimes.

Subset of Future Contracts

Follow these steps to get historical data for a subset of the Future contracts to which you are subscribed:

1. Call the GetFuturesContractList method with a Future Symbol and a datetimeDateTime to get a list of FutureContract Symbols that were trading at the given time.
   

var symbols = qb.FutureChainProvider.GetFutureContractList(symbol, startDate);

symbols = qb.FutureChainProvider.GetFutureContractList(symbol, start_date)

The GetFuturesContractList method returns a list of a few contracts Symbols.


2. (Optional) Create a subset of futures contracts from the full list:

var selectedSymbols = symbols
    .Where(s => FutureSymbol.IsStandard(s) && s.ID.Date <= startDate.AddDays(260));

selected_symbols = [s for s in symbols
    if FutureSymbol.IsStandard(s) and s.ID.Date <= start_date + timedelta(260)]

The code above mimics the filter specidied by the SetFilter method from the Create Subscriptions section.

3. If you want to get open interest data, call the AddFutureContract method with the FutureContract Symbol and disable fill-forward.

foreach (var symbol in selectedSymbols)
{
    qb.AddFutureContract(symbol, fillDataForward: false);
}

for symbol in selected_symbols:
    qb.AddFutureContract(symbol, fillDataForward = False)

4. Call the History method with a list of Symbols to get the historical data of the given Symbols.

var definedPeriodContractsHistory = qb.History(selectedSymbols, startTime, endTime);
var timespanContractsHistory = qb.History(selectedSymbols, TimeSpan.FromDays(10));
var barCountConstractsHistory = qb.History(selectedSymbols, 400);

var definedPeriodOpenInterestHistory = qb.History<OpenInterest>(selectedSymbols, startTime, endTime);
var timespanOpenInterestHistory = qb.History<OpenInterest>(selectedSymbols, TimeSpan.FromDays(10));
var barCountOpenInterestHistory = qb.History<OpenInterest>(selectedSymbols, 400);

defined_period_contracts_history = qb.History(selected_symbols, start_time, end_time)
timespan_contracts_history = qb.History(selected_symbols, timedelta(days=10))
bar_count_contracts_history = qb.History(selected_symbols, 400)

defined_period_open_interest_history = qb.History(OpenInterest, selected_symbols, start_time, end_time)
timespan_open_interest_history = qb.History(OpenInterest, selected_symbols, timedelta(days=10))
bar_count_open_interest_history = qb.History(OpenInterest, selected_symbols, 400)

You need some historical data to perform wrangling operations. Run a cell in a Jupyter Notebook with the pandas object as the last line to display the historical data.

The DataFrame that contains historical data for Futures contracts has a MultiIndex with levels for the expiry, and Symbol, and timestamp. For some of the wrangling operations, you need to drop the first index level. To drop the index level, run:

continuous_history.index = continuous_history.droplevel(0)
contract_history.index = contract_history.droplevel(0)
open_interest_history.index = open_interest_history.droplevel(0))

You need some historical data to perform wrangling operations. Use LINQ to wrangle the data and then call the Console.WriteLine method in a Jupyter Notebook to display the data.

Select One Contract

Iterate through the IEnumerable to get the historical data of the contract.

Index the DataFrame with a Symbol to select the historical data of the contract.

To display the price data of the contract, run:

foreach(var bar in contractHistory.TakeLast(5))
{
    Console.WriteLine($"{bar} EndTime: {bar.EndTime}");
}

contract_history.loc[symbol]

The Jupyter Notebook displays the trade data of the contract.

The Jupyter Notebook displays a DataFrame containing the trade and quote data of the contract.

To display the open interest data of the contract, run:

foreach(var openInterest in openInterestHistory.TakeLast(5))
{
    Console.WriteLine($"{openInterest} EndTime: {openInterest.EndTime}");
}

open_interest_history.loc[symbol]

The Jupyter Notebook displays the open interest of the contract.

The Jupyter Notebook displays a DataFrame containing the open interest of the contract.

Select a Property of the Contract

Iterate through the IEnumerable and select a property of the Slice to get the historical values of the property.

Index the DataFrame with a Symbol to select the historical data of the contract and then select a property column to get the historical values of the property.

To display the closing prices of the contract, run:

To display the closing ask prices of the contract, run:

var closingPrices = priceHistory.Select(data => $"{data.Symbol}: C: {data.Close}: EndTime: {data.EndTime}");
foreach(var closingPrices in closingPrices.TakeLast(5))
{
    Console.WriteLine(closingPrices);
}

contract_history.loc[symbol]['askclose']

The Jupyter Notebook displays the closing prices of the contract.

The Jupyter Notebook displays a Series containing the closing ask prices of the contract.

To display the open interest of the contract, run:

open_interest_history.loc[symbol]['openinterest']

The Jupyter Notebook displays a Series containing the open interest of the contract.

Unstack the Dataframe

If you request historical data for multiple contracts, you can transform the DataFrame so that it is a time series of bid closing prices for all of the containing contracts. Select the column that you want to display for each contract and then call the unstack method to transform the DataFrame into the desired format. You can rename the columns of the transformed DataFrame to be the respective FuturesContract Symbols.

To display the closing bid prices of all of the continuous contracts, run:

df = defined_period_continuous_history.copy()
df = df['bidclose'].unstack(level=0)
df.columns = [Symbol.GetAlias(SecurityIdentifier.Parse(x)) for x in df.columns]
df

The Jupyter Notebook displays a DataFrame that contains the closing bid prices of the continuous contracts and has the Symbol of each Futures as the column indices.

To display the closing bid prices of all of the contracts, run:

df = defined_period_contracts_history.copy()
df = df['bidclose'].unstack(level=0)
df.columns = [Symbol.GetAlias(SecurityIdentifier.Parse(x)) for x in df.columns]
df

The Jupyter Notebook displays a DataFrame that contains the closing bid prices of the contracts and has the Symbol of each FuturesContract as the column indices when we make a historical request with Symbol of FutureContract:

To display the open interest of all of the contracts, run:

df = timespan_open_interest_history.copy()
df = df['openinterest'].unstack(level=0)
df.columns = [Symbol.GetAlias(SecurityIdentifier.Parse(x)) for x in df.columns]
df

The Jupyter Notebook displays a DataFrame that contains the open interest of the contracts and has the Symbol of each FuturesContract as the column indices.

Jupyter Notebooks don't currently support libraries to plot historical data, but we are working on adding the functionality. Until we add the functionality, use Python to plot historical Futures data.

You need to get some historical Futures data to plot it. You can use many of the supported plotting libraries to visualize data in various formats. For example, you can plot candlestick and line charts.

Candlestick Chart

Follow these steps to plot candlestick charts:

1. Drop the first index level.

history.index = history.index.droplevel(0)

2. Select the contract Symbol index.

history = history.loc[symbol]

3. (Optional) Select the first 30 data points.

history = history.head(30)

4. Import the plotly library.

import plotly.graph_objects as go

5. Create a Candlestick.
   

candlestick = go.Candlestick(x=history.index,
                             open=history['open'],
                             high=history['high'],
                             low=history['low'],
                             close=history['close'])

6. Create a Layout.
   

layout = go.Layout(title=go.layout.Title(text='ES Continuous OHLC'),
                   xaxis_title='Date',
                   yaxis_title='Price',
                   xaxis_rangeslider_visible=False)

7. Create the Figure.
   

fig = go.Figure(data=[candlestick], layout=layout)

8. Show the Figure.
   

fig.show()

Candlestick charts display the open, high, low, and close prices of the contract.

Line Chart

Follow these steps to plot line charts using built-in methods:

1. Drop the first index level.

history.index = history.index.droplevel(0)

2. Select data to plot.

closing_prices = history['close'].unstack(level=0)

3. Rename the columns to be the Symbol of each contract.

closing_prices.columns = [Symbol.GetAlias(SecurityIdentifier.Parse(x)) for x in closing_prices.columns]

4. Call the plot method on the pandas object.

closing_prices.plot(title="Close", figsize=(15, 10))

5. Show the plot.

plt.show()

Line charts display the value of the property you selected in a time series.