Algorithm Reference

Historical Data

Introduction

QuantConnect allows you to request historical data in many different formats. In this section, we'll cover each of these formats and how you can use them in your algorithm. Broadly, there are two ways to use these data requests: direct historical data requests and indirect algorithm warm up.

A direct historical data request fetches any historical data you request, at any time for a given Symbol. This gives you the most control to apply the data as you need. Indirect algorithm warm up winds back the start time of your algorithm and feeds it with historical data, simulating a real data feed. Warm Up is best for quickly restoring the state of a fixed set of assets.

Below we'll explore these two options and how you can use them in your algorithm.

Key History Concepts

The QuantConnect historical data API has many different options to give you the greatest flexibility in how to apply it to your algorithm. Over the years it has evolved to handle different data formats, data resolutions and use cases, but we have always strived to keep two constants to its design:

Key Concept #1: Request Options
  • History requests can be done by a bar count, or,
  • History requests can be done for a period of time.

Secondly, when placing a history request it's important to consider what data will be returned as each asset type comes in slightly different data shapes. All python history requests return a Data Frame, which has different columns depending on the type of data requested. Data is returned as TradeBars, QuoteBars, or Slice objects depending on how you request it and the data available for your security.

Key Concept #2: Return Format
  • Single security requests return data as an array of the security data type. E.g. TradeBar[].
  • Multi-security (batch) requests return data as an array of Slice objects.
  • History requests with symbols provided return data as a pandas data frame.
  • History requests without symbols provided fetch history for the entire universe of your securities and return it as an array of Slice objects.

Finally, when reviewing the results of your history requests remember they are indexed by the EndTime of each data point. For daily data, this results in makes it appear datapoints appear on Saturday, and skipping Monday's bar.

Key Concept #3: Time Index
  • History results are indexed by the EndTime of a bar.

This can seem a little complex at first but we'll step through each of these in the sections below.

History Request For Known Symbols

The simplest form of history request is for a known set of Symbols. This is common for fixed universes of securities, or when you'd like to prepare new securities added to your algorithm. History requests return slightly different data depending on the overload you reach.

History data is returned in an ascending order from oldest to newest. This is required for piping the data into indicators to perform warm up.

History requests for a known set of Symbols return a Data Frame. Each row of the Data Frame represents the prices at a point of time. Each column of the Data Frame is a property of that price data (e.g. open, high, low, close).

Single Symbol Request Examples

History API calls follow the following pattern: self.History(symbol, bar_count, resolution = null)

# EXAMPLE 1: Requesting By Bar Count: 5 IBM TradeBars, defaulting to security resolution:
self.AddEquity("IBM", Resolution.Daily)
self.df = self.History(self.Symbol("IBM"), 5)

# EXAMPLE 2: Requesting By Bar Count: 5 IBM Minute TradeBars:
self.df = self.History(self.Symbol("IBM"), 5, Resolution.Minute)

# EXAMPLE 3: Requesting By Period: 1 Week IBM TradeBars, defaulting to security resolution: 
self.df = self.History(self.Symbol("IBM"), timedelta(7)) 

# Imporant Note: April 19th is Easter Friday, which has a bar EndTime = 20th, is not present.
# EXAMPLE 4: Requesting By Period: 5 Minutes IBM TradeBars: 
self.df = self.History(self.Symbol("IBM"), timedelta(5), Resolution.Minute)

# Important Note: Period history requests are relative to "now" algorithm time. The example above would return 5 minute bars if requested *at* market close. If you wait for 16.05 it will return nothing.

Multiple Symbol Request Examples

To request history for multiple symbols at a time you need to pass an array of Symbol objects to the same API methods as above.

Multi-Symbol History API calls follow the following pattern: self.History( symbols[], bar_count, resolution = null )

# EXAMPLE 5: Multi-Symbol History Request.

self.df = self.History([self.Symbol("IBM"), self.Symbol("AAPL")], 2)

Single Symbol History Request

When requesting a single Symbol LEAN can infer the type of the returning data is fixed and directly returns an array of bars. This is convient for quickly warming up an indicator.

// Single Symbol History Method Arguments: 
var bars = History<Type>(Symbol symbol, int barCount, Resolution resolution = null);
var bars = History<Type>(Symbol symbol, TimeSpan period, Resolution = null);
// EXAMPLE 1: 100 Bars of Single Symbol, Specifying Type, Default to Security Resolution:
var ibm = AddEquity("IBM", Resolution.Minute).Symbol;
var bars = History<TradeBar>(ibm, 100);

// Same request but for QuoteBars
var eurusd = AddForex("EURUSD", Resolution.Minute).Symbol;
var quoteBars = History<QuoteBar>(eurusd, 100);
// EXAMPLE 2: Six Hours of Bars of Single Symbol, Setting Resolution:
var ibm = AddEquity("IBM", Resolution.Minute).Symbol;
var bars = History<TradeBar>(ibm, TimeSpan.FromHours(6), Resolution.Minute);

// Same request but for QuoteBars
var eurusd = AddForex("EURUSD", Resolution.Minute).Symbol;
var quoteBars = History<QuoteBar>(eurusd, TimeSpan.FromHours(6), Resolution.Minute);

Multiple Symbols History Request

History requests with an array of Symbols objects return an array of slices. This provides a container which supports all data types. For example, a history request for FX-QuoteBars and Equity-TradeBars has the FX data is located under slices.QuoteBars, and the Equity data is under slices.Bars.

// Multi-Symbol History Method Arguments:
var slices = History(symbols[], int barCount, Resolution resolution = null);
var slices = History(Symbol[] symbols, TimeSpan period, Resolution = null);
// EXAMPLE 1: Symbol array, 5 Bars, Default to Security Resolution:
var ibm = AddEquity("IBM", Resolution.Minute).Symbol;
var aapl = AddEquity("AAPL", Resolution.Minute).Symbol;
var bars = History(new[] { ibm, aapl }, 5);
// EXAMPLE 2: Symbol array, 6 Hours, Specifying Resolution:
var ibm = AddEquity("IBM", Resolution.Minute).Symbol;
var aapl = AddEquity("AAPL", Resolution.Minute).Symbol;
var bars = History(new[] { ibm, aapl }, TimeSpan.FromHours(6), Resolution.Minute);

Assumed Default Values

  • Resolution. An important distinction exists between the two examples above, one is using the default value for Resolution. Here LEAN attempts to guess the resolution you request by looking at any securities you already have in your algorithm. If you have a matching Symbol, QuantConnect will use the same resolution. When no default values can be located Resolution.Minute is selected.
  • When no type is specified for the history request TradeBar is assumed for Equity, Futures, Crypto and Options securities. Assets with QuoteBar data must explicitly specify Quotes to receive their history (Forex, Futures, Options, and Crypto).

All Securities History Request

With the QuantConnect History API, you can request history for all active securities in your universe. The parameters are very similar to other history methods, but the return type is an array of Slice objects. This has the same properties as the OnData() Slice object.

The Slice object holds all of the results in a sorted enumerable collection you can iterate over with a foreach loop.

 
# EXAMPLE 1: Requesting 5 Bars For All Securities, default to security resolution:

# Setting Up Universe
self.AddEquity("IBM", Resolution.Daily)
self.AddEquity("AAPL", Resolution.Daily)

# Request history data and enumerate results:
self.slices = self.History(5)
for s in slices:
    print(str(s.Time) + \
          " AAPL:" + str(s.Bars["AAPL"].Close) + " IBM:" + str(s.Bars["IBM"].Close))

 
# EXAMPLE 2: Requesting 5 Minutes For All Securities:

slices = self.History(timedelta(minutes=5), Resolution.Minute)
for s in slices:
    print(str(s.Time) + \
          " AAPL:" + str(s.Bars["AAPL"].Close) + " IBM:" + str(s.Bars["IBM"].Close))

# Keep in mind your timedelta history requests are relative to "now" in Algorithm Time. If you requested this data at 16.05, it would return an empty array because the market was closed.
// EXAMPLE 1: Requesting 5 Bars For All Securities, default to security resolution:

// Setting Up Universe
AddEquity("IBM", Resolution.Daily)
AddEquity("AAPL", Resolution.Daily)

// Request history data and enumerate results:
var slices = History(5)
foreach (var s in slices) {
    Debug($"{s.Time} AAPL: {s.Bars["AAPL"].Close} IBM: {s.Bars["IBM"].Close}");
}

 
// EXAMPLE 2: Requesting 24 Hours of Hourly Data For All Securities:

var slices = History(TimeSpan.FromHours(24), Resolution.Hour);
foreach (var s in slices) {
     Debug($"{s.Time} AAPL: {s.Bars["AAPL"].Close} IBM: {s.Bars["IBM"].Close}");
}

// Keep in mind you TimeSpan history requests are relative to "now" in Algorithm Time. If you requested this data on a Monday morning, it would return an empty array because the market was closed over the weekend.

Working with Data Frames

In Python data is returned as Pandas DataFrame. It is a multi-index dataframe where the first index is the symbol. The data is then sorted in rows according to the EndTime of the bar. By learning a few helpful short-cuts you can directly access the history values you need for your algorithm.

# Setup Universe:
eurusd = self.AddForex("EURUSD", Resolution.Daily).Symbol
nzdusd = self.AddForex("NZDUSD", Resolution.Daily).Symbol

 # STEP 1:  Request Dataframe: 

self.df = self.History([eurusd, nzdusd], 3)



 # STEP 2: Check if empty and lock onto a symbol index with the loc[] method: 

if not self.df.empty:
    eurusd_quotebars = self.df.loc["EURUSD"]



# STEP 3: Extract and manipulate a single column with the string column name:

spread = eurusd_quotebars["askclose"] - eurusd_quotebars["bidclose"]

# Make sure to use the lowercase string column name.

Its possible to also perform analysis between symbols by unstacking the DataFrame. This will transform each column into the symbol values for one of the price-columns you select.

# UNSTACKING: Transform into columns:

# Fetch multi-indexed history:
self.dataframe = self.History([self.Symbol("IBM"), self.Symbol("AAPL")], 3)


# Transform using unstack:
self.dataframe["close"].unstack(level=0)

# Make sure to use the lowercase string column name.

History Data Formats

The QuantConnect platform hosts a specific set of data so the history data is limited to a few specific formats. See the table below for a guide to the format of data by security type:

Security Type Supported Types
EquityTradeBar
Tick, Second, Minute, Hour, Daily
Forex, CFDQuoteBar
Tick, Second, Minute, Hour, Daily
CryptoTradeBar (default), QuoteBar (available)
Tick, Second, Minute, Hour, Daily
Future ContractsTradeBar (default), QuoteBar (available)
Tick, Second, Minute
Option ContractsTradeBar (default), QuoteBar (available)
Minute

History is returned in TradeBars by default, but for Futures, Crypto and Options QuoteBars are also available. To request data as QuoteBars you must specify the type in the query.

// Get BTCUSD symbol and use it to request history 
var btcusd = AddCrypto("BTCUSD", Resolution.Daily, Market.GDAX).Symbol;
var quotes = History<QuoteBar>(btcusd, 14, Resolution.Daily)

Streaming Warm Up Period

In addition to the methods for manually requesting history above, QuantConnect also supports an automated "fast-forward" system called "Warm Up" which simulates winding back the clock from the time the algorithm is deployed. In a backest this is the StartDate of your algorithm. Warmup is a great way to prepare indicators for relatively simple strategies, but if you have a dynamic universe of assets we recommend manually requesting historical data when required.

The Warm Up API supports a set number of bars, or a period based warm up. Warmup should be called in your Initialize() method. An example of using warm up can be found here.

 // Wind time back 7 days from start:
SetWarmup(TimeSpan.FromDays(7));

// Feed in 100 bars before start date:
SetWarmup(100);
 # Wind time back 7 days from start:
self.SetWarmup(timedelta(7))

# Feed in 100 bars before start date:
self.SetWarmup(100)

Distinguishing Warmup from Reality

Your algorithm may need to distinguish warm up data from real data. QuantConnect makes this possible with a boolean flag IsWarmingUp self.IsWarmingUp. A common application of this flag might look like this:

// In Initialize
var emaFast = EMA("IBM", 50);
var emaSlow = EMA("IBM", 100);
SetWarmup(100);

// In OnData: Don't run if we're warming up our indicators.
if (IsWarmingUp) return;
# In Initialize
self.emaFast = self.EMA("IBM", 50);
self.emaSlow = self.EMA("IBM", 100);
self.SetWarmup(100);

// In OnData: Don't run if we're warming up our indicators.
if self.IsWarmingUp: return

Warmup Limitations

Algorithm Warmup is useful if you have a streaming algorithm which can incrementally build algorithm state however it has several limitations:

  • Trades cannot be performed during warm up as they would impact the algorithm portfolio and would be trading on fictional fast-forwarded data.
  • Due to technical limitations Universe selection cannot be fast-forwarded. Any universe selection is skipped until realtime is reached.

You can also see our Tutorials and Videos. You can also get in touch with us via Chat.

Did you find this page Helpful ?