Back

[Feature] History and Warmup

Hey all!

I'm here to publicly announce a feature that we've all been waiting for, history and warmup! With these new features you can request historical data from any point in the algorithm. You can also specify a warm up period where we'll pump historical data through the algorithm to get it into a ready state. This is great for live trading when you have some indicators that may take weeks, months, or even years to reach a ready state. Now that same algorithm can be ready to start trading within minutes of deploying!

Here's a link to all the History and Warmup API methods on the base algorithm class.

Have a look at the simple history demo algorithm I put together for you guys. Feel free to ask questions!
Update Backtest






The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Michael,

The History functionality is very useful.
While trying to use it, I met some difficulty in getting minute bar history while subscribing to per tick data. Please see the code attached.
Is it because we can only get history for subscribed resolution?

Also want to ask if the history function is available for indicators? What can I do to get the previous RSI value formed by the previous bars?

Thanks a lot!

Jianwei
0


History function should work on any resolution you've requested, I'll look into it with Mike to find out why its not working. In the future please post bug reports to https://www.quantconnect.com/support

History returns an array of data and you can use this to manually warm up your indicators, or you can also use "SetWarmup(timespan)" to pump data through the OnData and indicator update methods like normal. SetWarmup sets a period before the start date and starts pumping in data early, then it also does the same in live mode.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Will do next time.
Thank you.
0

Hey, the tick bug has been recently fixed and will be available with the next deploy, thanks to @StefanoRaggi for the fix!
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thank you! Looking forward to the release
0

How does the History call behave when you're running in Live mode? I assume the same?
0

In live mode custom data uses exactly the same code, so it should behave 100% the same. Non custom data (ticks/trade bars) will make a request over the internet for the requested historical data, so it may take just a little bit longer depending on the size of the request and how frequently you would like to do it.

Other than the potential delay inherent in running in a live environment, they should behave the same!
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


This is awesome! I've been having trouble implementing it in my forex, GBP/USD, algo. Does it work with forex data? Is there any significant changes to the code?
0

Hey Timothy, it works with any data type. Please post an example algorithm so we can help you further.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


its saying there is an error with the foreach(var tradeBar in History) line
// Set Warmup
int warmupPeriod = emaMultiple*emaSlow;
var history = History(Symbol, warmupPeriod);
foreach (var tradeBar in history)
{
fast.Update(tradeBar.EndTime, tradeBar.Close);
slow.Update(tradeBar.EndTime, tradeBar.Close);
confirmation.Update(tradeBar.EndTime, tradeBar.Close);
}
0

I believe its all working fine Tim. Please share a whole algorithm if you think there's a specific error.
See backtest attached demonstrating EURUSD history.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


I appreciate the help however Im hesitant to post my algo for privacy reasons. Does this error mean anything to anyone?

Runtime Error: This is a forward only indicator: EMA150(GBPUSD_min) Input: 2015-10-23 00:22:00Z Previous: 2015-10-27 19:01:00Z (Open Stacktrace)
0

Hey Timothy, without the code it's hard to be sure, but it looks like you're feeding in multiple data streams to the same indicator. Possibly passing history to an indicator AND using the warm up feature? This error is a warning saying that you're passing past data into an indicator. In this case, you've passed data into the EMA_150(GBPUSD_min) indicator up until 2015.10.27 19:01:00Z, but then you tried to pass data from 2015.10.23 00:22:00Z - So the indicator throws an exception since you're not feeding it a continuous stream of data and would give spurious results.

If you're concerned about privacy, it's typically fairly easy to clone the algorithm and remove and intellectual property you'd like to keep private. I hope this description of the problem helps. If you're still having issues, try to reproduce it in an algorithm you're willing to share.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Oh this is fantastic. Thanks Michael!
0

Hi,
When I try to run Jared's example above I get:

Runtime Error: 'EURUSD' wasn't found in the TradeBars object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("EURUSD") (Open Stacktrace)

Could you please check what is going on? I got similar error running another forex example and this issue seems to be a recent one. Thanks.
0

Its always good practice to read the stack trace (by clicking the link in the error message). The error is that the benchmark feed (SPY) is getting through to the OnData method, and is fixed by the line mentioned in the error message:
if (!data.ContainsKey("EURUSD")) return;

Or you can set a custom benchmark so your statistics are relative to EURUSD (after AddSecurity):
SetBenchmark("EURUSD");
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Jared,

Quick question: if I have coarse universe resolution set to "Minute", and I need to warmup the universe, say, for 5 days (which is 1950 tradeable minutes), would I need to do the following:

SetWarmup(1950)

or is this function expecting "days" as the period, as the universe only rotates daily?

Thanks!

0

Hey Stephen, how are you trying to use warm up exactly? Warm up runs exactly once right after initialization. If you're using universe selection and didn't manually add securities then there's nothing to 'warm-up'. If you need to get data for securities added via universe selection, have a peek at the history api.

To answer your question, the periods in there is specific to each security. For example, say I have VXX minute and SPY daily and SetWarmup(100), this will get me 100 minutes points for VXX and 100 daily points for SPY. This history function behaves the same way, so you could request the last 100 data points of history for all securities using something like the following:var slices = History(100);Slices here is an IEnumerable<Slice>

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Michael,

Sorry I wasn't clear earlier; I'm trying to use SetWarmup to warm up the indicators I use in my coarse universe filter (similar to the SelectionData class you implemented in the Cross EMA coarse universe example algorithm). Else, live trading the program takes several days to actually start crunching.

I think I may be using SetWarmup incorrectly, and may need to use history as you suggest. I'm getting the feeling that all warmup is, is running the program for a predefined period prior to the official start date. This is VERY useful for backtesting, but I'm guessing it doesn't translate to live trading very well. :-P

0

Sadly universe selection is not currently involved in the warm up period. On the first invocation of the coarse selection function you could gather all of the symbols and make one massive history request for it using the history overload that accepts symbols and a period count. Warm up does work in live trading, but only for securities that have already been added to the algorithm. In universe selection, you're scanning securities that have not been added, and so warm up won't know to get data for these guys.

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed