Daily Data Backtesting and Yahoo Data by Gambulator

Gambulator INVESTOR

| Reward Discussion

(1) I am new to this platform but know C# and Java... If I want to access daily data, do I have to do the aggregation myself ? I see other postings and it seems like that's what I need now. or there is something like AddSecurity(SecurityType.Equity, symbol, Resolution.Daily??); resolution can be daily ?
(2) Perhaps I should clone a few algo but how do I access for say close price for min bar and 1 min before ? like close[bar] and close[bar-1] in other platforms ?

Update Backtest

Gambulator

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Jared Broad

Welcome @gambulator! -
1) We will add in the Resolution.Daily very soon - its been on the to-do list for for too long :)
2) We don't have anything like that built in but will think about it next time we update the API. For now its fairly simple to do with a line of code:



TradeBars previous = new TradeBars();

public void OnData(TradeBars bars) 

{

   //Do stuff. then save the bar to the class variable.

   previous = bars;

}

Michael H

35.4k ,

You can use something like the following to get similar syntax as with other platforms. This type is used extensively in the Composable Indicators project attached.

You would use the code below as follows:



RollingWindow window = new RollingWindow(10); // create a 10 day window



public override void OnTradeBar(Dictionary data)

{

    TradeBar bar;

    if (data.TryGetValue("SPY", out bar))

    {

        window.Update(bar.Close);

        if (window.IsReady)

        {

             decimal tenDayChange = window[0] - window[9];

        }

    }

}



using System;

using System.Collections;

using System.Collections.Generic;



namespace QuantConnect

{

    /// 

    ///     Interface type used to pass windows around without worry of external modification

    /// 


    /// The type of data in the window

    public interface IReadOnlyWindow : IEnumerable

    {

        /// 

        ///     Gets the size of this window

        /// 


        int Size { get; }



        /// 

        ///     Gets the current number of elements in this window

        /// 


        int Count { get; }



        /// 

        ///     Gets the number of samples that have been added to this window over its lifetime

        /// 


        decimal Samples { get; }

        

        /// 

        ///     Indexes into this window, where index 0 is the most recently

        ///     entered value

        /// 


        /// the index, i

        /// the ith most recent entry

        T this[int i] { get; }



        /// 

        ///     Gets a value indicating whether or not this window is ready, i.e,

        ///     it has been filled to its capacity, this is when the Size==Count

        /// 


        bool IsReady { get; }



        /// 

        /// Gets the most recently removed item from the window. This is the

        /// piece of data that just 'fell off' as a result of the most recent

        /// add. If no items have been removed, this will throw an exception.

        /// 


        T MostRecentlyRemoved { get; }

    }



    /// 

    ///     This is a window that allows for list access semantics,

    ///     where this[0] refers to the most recent item in the

    ///     window and this[Count-1] refers to the last item in the window

    /// 


    /// The type of data in the window

    public class RollingWindow : IReadOnlyWindow

    {

        private int _tail;

        private decimal _samples;

        private T _mostRecentlyRemoved;

        private readonly List _list;

        private readonly object _lock = new object();



        /// 

        ///     Gets the size of this window

        /// 


        public int Size

        {

            get { return _list.Capacity; }

        }



        /// 

        ///     Gets the current number of elements in this window

        /// 


        public int Count

        {

            get { return _list.Count; }

        }



        /// 

        ///     Gets the number of samples that have been added to this window over its lifetime

        /// 


        public decimal Samples

        {

            get { return _samples; }

        }



        public T MostRecentlyRemoved

        {

            get

            {

                if (!IsReady)

                {

                    throw new InvalidOperationException("No items have been removed yet!");

                }

                return _mostRecentlyRemoved;

            }

        }



        public RollingWindow(int size)

        {

            _list = new List(size);

        }



        /// 

        ///     Indexes into this window, where index 0 is the most recently

        ///     entered value

        /// 


        /// the index, i

        /// the ith most recent entry

        public T this[int i]

        {

            get

            {

                if (i >= Count)

                {

                    throw new ArgumentOutOfRangeException("i", i, string.Format("Must be between 0 and Count {{{0}}}", Count));

                }

                return _list[(Count + _tail - i - 1)%Count];

            }

            set

            {

                if (i >= Count)

                {

                    throw new ArgumentOutOfRangeException("i", i, string.Format("Must be between 0 and Count {{{0}}}", Count));

                }

                _list[(Count + _tail - i - 1) % Count] = value;

            }

        }



        /// 

        ///     Gets a value indicating whether or not this window is ready, i.e,

        ///     it has been filled to its capacity, this is when the Size==Count

        /// 


        public bool IsReady

        {

            get { return Samples > Count; }

        }



        /// 

        ///     Adds an item to this window and shifts all other elements

        /// 


        /// The item to be added

        public void Add(T item)

        {

            lock (_lock)

            {

                _samples++;

                if (Size == Count)

                {

                    // keep track of what's the last element

                    // so we can reindex on this[ int ]

                    _mostRecentlyRemoved = _list[_tail];

                    _list[_tail] = item;

                    _tail = (_tail + 1)%Size;

                }

                else

                {

                    _list.Add(item);

                }

            }

        }



        /// 

        ///     Clears this window of all data

        /// 


        public void Clear()

        {

            lock (_lock)

            {

                _list.Clear();

            }

        }



        public IEnumerator GetEnumerator()

        {

            // we make a copy on purpose so the enumerator isn't tied 

            // to a mutable object, well it is still mutable but out of scope

            var temp = new List(Count);

            lock (_lock)

            {

                for (int i = 0; i < Count; i++)

                {

                    temp.Add(this[i]);

                }

            }

            return temp.GetEnumerator();

        }



        IEnumerator IEnumerable.GetEnumerator()

        {

            return GetEnumerator();

        }

    }

}

Kevin Roe

2.1k ,

Would it be (or is it now) possible to use multiple tradebar resolutions at the same time?

Something like:

public void OnDailyData(TradeBars data) {...}
public void OnMinuteData(TradeBars data) {...}
public void OnSecondData(TradeBars data) {...}

Jared Broad

No sorry, but you could import Yahoo daily data as a Custom data source e.g. ("SPY_YAHOO"), and use QC's minute data for the execution ("SPY"). You'd just add them both as data sources:

AddSecurity(SecurityType.Equity, "SPY", Resolution.Minute); //Will fire every minute

AddData("SPY_YAHOO", Resolution.Minute); // Will fire at start of day with full OHLC.

I've attached an algorithm which does exactly this, using Quandl as a wrapper for Yahoo to simplify the data-sorting. I purchase "SPY_DAILY" and it runs the backtest using Yahoo/Quandl prices.

Check out the Yahoo.cs file for how I imported custom data --

The bars are time-stamped midnight (because they're daily). Be careful you don't use look ahead bias with the day's closing price... :)

Kevin Roe

2.1k ,

Now we're getting somewhere! Thanks Jared, this is very useful.

Hakan Deveci

159 ,

Hey Jared,

What if we want to add more than one custom price data? Do we have to create a class for each symbol or is there a way to generalize your Yahoo.cs class? Say, I want to use something like this:

AddData("SPY_DAILY", Resolution.Minute);

AddData("VIX_DAILY", Resolution.Minute);

AddData("TCELL_DAILY", Resolution.Minute);

Jared Broad

Hey Hakan! Yes the "config" value passes in the symbol, so with tweaking you can make it look up a different URL - I've hacked together a Google & Yahoo generic version here -

The drops to 0 in the chart are from the Google data - its pretty spotty. Perhaps you'll have a better source.

Gustavo is making a data manager now which will auto generate these data import scripts. We think it will be online by Friday this week -

Hakan Deveci

159 ,

That's great, thank you!

Nik milev

103.9k ,

Hi All,
This is an old post, but I stumbled on it when trying to import data from Yahoo for several symbols and run locally the Lean engine.
The way I understand it is that "public void OnData(Yahoo data)" will trigger only for SPY until all SPY data is exhausted and then trigger only for VIX.
Basically here we lost the bars synchronization for SPY and VIX, available in TradeBars dictionary (which we would have had if we ran it from the site without custom data importing).
Am I correct or I am missing something?
Also, are there are examples to have several custom read symbols ( e.g. SPY and VIX) and triggering OnData(TradeBars data) so we can take advantage of their synchronized data ?
Thanks
Nik

Jared Broad

True its not optimal for multiple symbols! I'll chat with @Michael about this on Monday. We're in the middle of a larger FOREX update but once its done we can review the method invokers. This is implemented here, and if made into a list it could send multiple objects in the same time slice or as a supplementary property on the TradeBars collection.

Michael Handschuh

50.5k ,

When running multiple custom symbols of the same type (such as Yahoo) then then OnData(Yahoo data) event will be fired once per symbol for each time step.

If I have the following in Initialize():

AddData("YAHOO/INDEX_SPY");

AddData("YAHOO/IBM");

The first data point will be for the "YAHOO/INDEX_SPY" symbol, and then the next event fired will be the YAHOO/IBM symbol. They will continue to alternate until they run out of data.

I've attached a project which uses two custom symbols and a Forex symbol.

Gambulator INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

Community Ambassadors

Earn free QuantConnect cloud credits by helping others, each week we airdrop credits to the most active community members.

232,981 Quants.

Online Now

Stay Connected