Back

Daily Data Backtesting and Yahoo Data

(1) I am new to this platform but know C# and Java... If I want to access daily data, do I have to do the aggregation myself ? I see other postings and it seems like that's what I need now. or there is something like AddSecurity(SecurityType.Equity, symbol, Resolution.Daily??); resolution can be daily ?
(2) Perhaps I should clone a few algo but how do I access for say close price for min bar and 1 min before ? like close[bar] and close[bar-1] in other platforms ?
Update Backtest








Welcome @gambulator! -
1) We will add in the Resolution.Daily very soon - its been on the to-do list for for too long :)
2) We don't have anything like that built in but will think about it next time we update the API. For now its fairly simple to do with a line of code:

TradeBars previous = new TradeBars();
public void OnData(TradeBars bars)
{
//Do stuff. then save the bar to the class variable.
previous = bars;
}
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


You can use something like the following to get similar syntax as with other platforms. This type is used extensively in the Composable Indicators project attached.

You would use the code below as follows:


RollingWindow window = new RollingWindow(10); // create a 10 day window

public override void OnTradeBar(Dictionary data)
{
TradeBar bar;
if (data.TryGetValue("SPY", out bar))
{
window.Update(bar.Close);
if (window.IsReady)
{
decimal tenDayChange = window[0] - window[9];
}
}
}




using System;
using System.Collections;
using System.Collections.Generic;

namespace QuantConnect
{
///
/// Interface type used to pass windows around without worry of external modification
///

/// The type of data in the window
public interface IReadOnlyWindow : IEnumerable
{
///
/// Gets the size of this window
///

int Size { get; }

///
/// Gets the current number of elements in this window
///

int Count { get; }

///
/// Gets the number of samples that have been added to this window over its lifetime
///

decimal Samples { get; }

///
/// Indexes into this window, where index 0 is the most recently
/// entered value
///

/// the index, i
/// the ith most recent entry
T this[int i] { get; }

///
/// Gets a value indicating whether or not this window is ready, i.e,
/// it has been filled to its capacity, this is when the Size==Count
///

bool IsReady { get; }

///
/// Gets the most recently removed item from the window. This is the
/// piece of data that just 'fell off' as a result of the most recent
/// add. If no items have been removed, this will throw an exception.
///

T MostRecentlyRemoved { get; }
}

///
/// This is a window that allows for list access semantics,
/// where this[0] refers to the most recent item in the
/// window and this[Count-1] refers to the last item in the window
///

/// The type of data in the window
public class RollingWindow : IReadOnlyWindow
{
private int _tail;
private decimal _samples;
private T _mostRecentlyRemoved;
private readonly List _list;
private readonly object _lock = new object();

///
/// Gets the size of this window
///

public int Size
{
get { return _list.Capacity; }
}

///
/// Gets the current number of elements in this window
///

public int Count
{
get { return _list.Count; }
}

///
/// Gets the number of samples that have been added to this window over its lifetime
///

public decimal Samples
{
get { return _samples; }
}

public T MostRecentlyRemoved
{
get
{
if (!IsReady)
{
throw new InvalidOperationException("No items have been removed yet!");
}
return _mostRecentlyRemoved;
}
}

public RollingWindow(int size)
{
_list = new List(size);
}

///
/// Indexes into this window, where index 0 is the most recently
/// entered value
///

/// the index, i
/// the ith most recent entry
public T this[int i]
{
get
{
if (i >= Count)
{
throw new ArgumentOutOfRangeException("i", i, string.Format("Must be between 0 and Count {{{0}}}", Count));
}
return _list[(Count + _tail - i - 1)%Count];
}
set
{
if (i >= Count)
{
throw new ArgumentOutOfRangeException("i", i, string.Format("Must be between 0 and Count {{{0}}}", Count));
}
_list[(Count + _tail - i - 1) % Count] = value;
}
}

///
/// Gets a value indicating whether or not this window is ready, i.e,
/// it has been filled to its capacity, this is when the Size==Count
///

public bool IsReady
{
get { return Samples > Count; }
}

///
/// Adds an item to this window and shifts all other elements
///

/// The item to be added
public void Add(T item)
{
lock (_lock)
{
_samples++;
if (Size == Count)
{
// keep track of what's the last element
// so we can reindex on this[ int ]
_mostRecentlyRemoved = _list[_tail];
_list[_tail] = item;
_tail = (_tail + 1)%Size;
}
else
{
_list.Add(item);
}
}
}

///
/// Clears this window of all data
///

public void Clear()
{
lock (_lock)
{
_list.Clear();
}
}

public IEnumerator GetEnumerator()
{
// we make a copy on purpose so the enumerator isn't tied
// to a mutable object, well it is still mutable but out of scope
var temp = new List(Count);
lock (_lock)
{
for (int i = 0; i < Count; i++)
{
temp.Add(this[i]);
}
}
return temp.GetEnumerator();
}

IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
}
0

Would it be (or is it now) possible to use multiple tradebar resolutions at the same time?

Something like:

public void OnDailyData(TradeBars data) {...}
public void OnMinuteData(TradeBars data) {...}
public void OnSecondData(TradeBars data) {...}
0

No sorry, but you could import Yahoo daily data as a Custom data source e.g. ("SPY_YAHOO"), and use QC's minute data for the execution ("SPY"). You'd just add them both as data sources:

AddSecurity(SecurityType.Equity, "SPY", Resolution.Minute); //Will fire every minute
AddData("SPY_YAHOO", Resolution.Minute); // Will fire at start of day with full OHLC.


I've attached an algorithm which does exactly this, using Quandl as a wrapper for Yahoo to simplify the data-sorting. I purchase "SPY_DAILY" and it runs the backtest using Yahoo/Quandl prices.

Check out the Yahoo.cs file for how I imported custom data --

The bars are time-stamped midnight (because they're daily). Be careful you don't use look ahead bias with the day's closing price... :)
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Now we're getting somewhere! Thanks Jared, this is very useful.
0

Hey Jared,

What if we want to add more than one custom price data? Do we have to create a class for each symbol or is there a way to generalize your Yahoo.cs class? Say, I want to use something like this:

AddData("SPY_DAILY", Resolution.Minute);
AddData("VIX_DAILY", Resolution.Minute);
AddData("TCELL_DAILY", Resolution.Minute);
0

Hey Hakan! Yes the "config" value passes in the symbol, so with tweaking you can make it look up a different URL - I've hacked together a Google & Yahoo generic version here -

The drops to 0 in the chart are from the Google data - its pretty spotty. Perhaps you'll have a better source.

Gustavo is making a data manager now which will auto generate these data import scripts. We think it will be online by Friday this week -
1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


That's great, thank you!
0

Hi All,
This is an old post, but I stumbled on it when trying to import data from Yahoo for several symbols and run locally the Lean engine.
The way I understand it is that "public void OnData(Yahoo data)" will trigger only for SPY until all SPY data is exhausted and then trigger only for VIX.
Basically here we lost the bars synchronization for SPY and VIX, available in TradeBars dictionary (which we would have had if we ran it from the site without custom data importing).
Am I correct or I am missing something?
Also, are there are examples to have several custom read symbols ( e.g. SPY and VIX) and triggering OnData(TradeBars data) so we can take advantage of their synchronized data ?
Thanks
Nik
0

True its not optimal for multiple symbols! I'll chat with @Michael about this on Monday. We're in the middle of a larger FOREX update but once its done we can review the method invokers. This is implemented here, and if made into a list it could send multiple objects in the same time slice or as a supplementary property on the TradeBars collection.
1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


When running multiple custom symbols of the same type (such as Yahoo) then then OnData(Yahoo data) event will be fired once per symbol for each time step.

If I have the following in Initialize():
AddData("YAHOO/INDEX_SPY");
AddData("YAHOO/IBM");


The first data point will be for the "YAHOO/INDEX_SPY" symbol, and then the next event fired will be the YAHOO/IBM symbol. They will continue to alternate until they run out of data.

I've attached a project which uses two custom symbols and a Forex symbol.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed