Consolidator for custom data by Greg L

Greg L INVESTOR

| Reward Discussion

Hi,

I have a custom data type extending BaseData, this data is minute by minute and I'd like to consolidate the bars into hourly bars. I've been looking at the examples but I have not been able to figure this out just yet.

Does anyone have a sample of something similar?

Also fwiw I am running the Lean engine via VS2013 Community Ed. on my Windows 8.1 workstation and the custom data is a csv on the workstation.

Thanks,
Greg L.

Update Backtest

Greg L

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Michael Handschuh

50.5k ,

Hey Greg!

Can you post your custom type? I typically just use the built in resolver:



var consolidator = ResolveConsolidator("CustomSymbol", Resolution.Hour);

Greg L

444 ,

Hi Michael,

Thanks for the help, I didn't know to use the ResolveConsolidator - that works great for the hourly bars. Should



var consolidator = ResolveConsolidator("CustomSymbol", Resolution.Daily );

work the same? From my test it did not seem to. When its set to Resolution.Hour the HourBarHandler event is triggered and the Console.Writeline happens.

If I change Resolution.Hour to Resolution.Daily the event does not seem to fire.

Here is my code:



 public class TestingCustomDataAlgo : QCAlgorithm

    {



        /// 

        /// Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.

        /// 


        public override void Initialize()

        {

          

            SetStartDate(new DateTime(2010, 04, 02, 00, 00, 00));

            SetEndDate(2010, 05, 04);



            AddData("EURUSD");



            //wire up an event to handle the bars

            var consolidator = ResolveConsolidator("EURUSD", Resolution.Hour);

            //var consolidator = ResolveConsolidator("EURUSD", Resolution.Daily);

            consolidator.DataConsolidated += HourBarHandler;

            SubscriptionManager.AddConsolidator("EURUSD", consolidator);



        }              



        /// 

        /// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.

        /// 


        /// TradeBars IDictionary object with your stock data

        public void OnData(EURUSD data)

        {

            //Console.WriteLine("Time: " + data.Time);           

        }



        private void HourBarHandler(object sender, BaseData consolidated)

        {

            Console.WriteLine("Consolidated Time: " + consolidated.Time);

        }

  

    }



/// 

    /// EURUSD is a custom data type we create for this algorithm

    /// 


    public class EURUSD : BaseData

    {

        /// 

        /// Opening Price

        /// 


        public decimal Open = 0;

        /// 

        /// High Price

        /// 


        public decimal High = 0;

        /// 

        /// Low Price

        /// 


        public decimal Low = 0;

        /// 

        /// Closing Price

        /// 


        public decimal Close = 0;

       

        /// 

        /// Default initializer for EURUSD.

        /// 


        public EURUSD()

        {

            Symbol = "EURUSD";

        }



        /// 

        /// Return the URL string source of the file. This will be converted to a stream 

        /// 


        public override string GetSource(SubscriptionDataConfig config, DateTime date, DataFeedEndpoint datafeed)

        {

            return @"..\..\..\Data\custom\EURUSD_1M_20100401_20141111.csv";

        }



        /// 

        /// Reader converts each line of the data source into BaseData objects. Each data type creates its own factory method, and returns a new instance of the object 

        /// each time it is called. 

        /// 


        public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, DataFeedEndpoint datafeed)

        {

           

            EURUSD index = new EURUSD();



            try

            {

                string[] data = line.Split(',');

                string timeValue = (Convert.ToInt32(data[2])).ToString("00:00:00");

                string dateValue = (Convert.ToInt32(data[1])).ToString("0000/00/00");

                index.Symbol = data[0];

                index.Time = DateTime.Parse(dateValue + " " + timeValue);

                index.Open = Convert.ToDecimal(data[3], CultureInfo.InvariantCulture);

                index.High = Convert.ToDecimal(data[4], CultureInfo.InvariantCulture);

                index.Low = Convert.ToDecimal(data[5], CultureInfo.InvariantCulture);

                index.Close = Convert.ToDecimal(data[6], CultureInfo.InvariantCulture);

                index.Value = index.Close;

            }

            catch (Exception ex)

            {

                ex.ToString();

            }



            return index;

        }



    }

Basically instead of re-formatting / packaging my csv data I found it easier to manipulate the time and date fields using the custom type Reader method.

Thanks again for the help,
Greg

Michael Handschuh

50.5k ,

Hey Greg,

I've attached an example running against custom bitcoin data that uses the daily consolidation. This seems to work as expected. I'm unable to run your exact configuration since it looks like you're running LEAN locally against some custom data. If you'd like, put your custom data in dropbox and we can run it here on the cloud to further diagnose your issue.

Greg L

444 ,

Hi Michael,

Thanks for the help, I'll tinker with it for a bit and see if I can figure out - if that fails I'll do a dropbox link and let you know.

Appreciate your time!
Greg

Greg L

444 ,

Hi Michael, The daily resolution does work - I just put my test code in the wrong method. I am running into another issue that's related to the data itself, the fact that its forex data to be precise. Because the data contains an abbreviated Sunday session the daily bar ends for this at 21:15 and what I am finding is that this seems to roll forward into all the daily bars after the first Sunday. This is a sample from my testing:

Consolidated Time: Wednesday at 5/5/2010 12:00:00 AM
Consolidated Time: Thursday at 5/6/2010 12:00:00 AM
Consolidated Time: Friday at 5/7/2010 12:00:00 AM
Consolidated Time: Sunday at 5/9/2010 9:15:00 PM
Consolidated Time: Monday at 5/10/2010 9:15:00 PM

Would you have any thoughts/ideas on how I could get the days following a Sunday to start using 00:00 (aka 12:00AM)?

Thanks again for your time,
Greg

Michael Handschuh

50.5k ,

Hey Greg, I'll submit a bug report to the LEAN github. This is certainly not the desired behavior :).

I've attached a project that defines a DailyTradeBarConsolidator. This should behave closer to as expected:
2015-03-02 00:01:00 Consolidated Time: Sunday - 03/01/2015 19:30:00
2015-03-03 00:01:00 Consolidated Time: Monday - 03/02/2015 00:00:00
2015-03-04 00:01:00 Consolidated Time: Tuesday - 03/03/2015 00:00:00
2015-03-05 00:01:00 Consolidated Time: Wednesday - 03/04/2015 00:00:00
2015-03-06 00:01:00 Consolidated Time: Thursday - 03/05/2015 00:00:00
2015-03-08 17:01:00 Consolidated Time: Friday - 03/06/2015 00:00:00
2015-03-09 00:01:00 Consolidated Time: Sunday - 03/08/2015 17:00:00
2015-03-10 00:01:00 Consolidated Time: Monday - 03/09/2015 00:00:00
2015-03-11 00:01:00 Consolidated Time: Tuesday - 03/10/2015 00:00:00
2015-03-12 00:01:00 Consolidated Time: Wednesday - 03/11/2015 00:00:00
2015-03-13 00:01:00 Consolidated Time: Thursday - 03/12/2015 00:00:00
2015-03-15 17:01:00 Consolidated Time: Friday - 03/13/2015 00:00:00
2015-03-16 00:01:00 Consolidated Time: Sunday - 03/15/2015 17:00:00
2015-03-17 00:01:00 Consolidated Time: Monday - 03/16/2015 00:00:00

I'll work on implementing a more permanent fix to this later this week. Let me know if this fits your needs!

Greg L

444 ,

Thanks Michael, I looked at this briefly and it seems like it will suit my needs just fine. Also thanks for submitting the issue with the LEAN development team.

Thanks again for your help!
Greg

Greg L

444 ,

Hi Michael,

When I use the DailyTradeBarConsolidator this way



var dtbConsolidator = new DailyTradeBarConsolidator();

dtbConsolidator.DataConsolidated += DayBarHandler;

SubscriptionManager.AddConsolidator("EURUSD", dtbConsolidator);

I get a runtime error like this:

Type mismatch found between consolidator and symbol. Symbol: EURUSD expects type EURUSD but tried to register consolidator with input type TradeBar

Conceptually I think I get difference between a 'symbol' and a 'bar' of that symbol but without actually reading all the source code I'm not seeing how to get a 'TradeBar' from 'BaseData.' Is there a method I can override to return a TradeBar from my type that extends BaseData or do I need to use a different approach?

Thanks again for your assistance,
Greg

Michael Handschuh

50.5k ,

Please attach the entire project, or a subset of the project showcasing the error. This looks like you're trying to use a custom data type called 'EURUSD' but the consolidator is expecting 'TradeBar' data.

Greg L

444 ,

Thanks again for your help! Your understanding is correct and that was my question to which I did not know the answer. Code is below:



using System;

using System.IO;

using System.Collections.Generic;

using System.Globalization;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

using QuantConnect.Data;

using QuantConnect.Data.Market;

using QuantConnect.Data.Consolidators;



namespace QuantConnect.Algorithm.Examples

{



    /// 

    /// Testing algorithm simply initializes the date range and cash

    /// 


    public class TestingCustomDataAlgo : QCAlgorithm

    {



        /// 

        /// Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.

        /// 


        public override void Initialize()

        {

          

            SetStartDate(new DateTime(2010, 05, 03, 00, 00, 00));

            SetEndDate(2010, 06, 01);



            AddData("EURUSD");



            //create an event to handle the hourly bars

            var consolidator = ResolveConsolidator("EURUSD", Resolution.Daily);

            consolidator.DataConsolidated += HourBarHandler;

            SubscriptionManager.AddConsolidator("EURUSD", consolidator);



            var dtbConsolidator = new DailyTradeBarConsolidator();

            dtbConsolidator.DataConsolidated += DayBarHandler;

            SubscriptionManager.AddConsolidator("EURUSD", dtbConsolidator);



        }



        private void DayBarHandler(object sender, BaseData consolidated)

        {

            Console.WriteLine("\nDBH Consolidated Time: " + consolidated.Time.DayOfWeek + " at " + consolidated.Time);

        }              



        /// 

        /// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.

        /// 


        /// TradeBars IDictionary object with your stock data

        public void OnData(EURUSD data)

        {

            //Console.WriteLine("Time: " + data.Time);           

        }



        private void HourBarHandler(object sender, BaseData consolidated)

        {

            Console.WriteLine("\nConsolidated Time: " + consolidated.Time.DayOfWeek + " at " + consolidated.Time);



            //Debug("\nConsolidated Daily Time: " + consolidated.Time);

        }  

    }



    /// 

    ///  ad hoc fix for daily close issue

    /// 


    public class DailyTradeBarConsolidator : DataConsolidator

    {

        private TradeBar _working;



        /// 

        /// OutputType returns a Type?

        /// 


        public override Type OutputType

        {

            get { return typeof(TradeBar); }

        }



        /// 

        /// Update method

        /// 


        /// Tradebar type

        public override void Update(TradeBar data)

        {

            if (_working != null && _working.Time.Date != data.Time.Date)

            {

                OnDataConsolidated(_working);

                _working = null;

            }



            AggregateBar(ref _working, data);

        }



        /// 

        /// Aggregates the new 'data' into the 'workingBar'. The 'workingBar' will be

        /// null following the event firing

        /// 


        /// The bar we're building, null if the event was just fired and we're starting a new trade bar

        /// The new data

        protected void AggregateBar(ref TradeBar workingBar, TradeBar data)

        {

            if (workingBar == null)

            {

                workingBar = new TradeBar

                {

                    Time = data.Time,

                    Symbol = data.Symbol,

                    Open = data.Open,

                    High = data.High,

                    Low = data.Low,

                    Close = data.Close,

                    Volume = data.Volume,

                    DataType = MarketDataType.TradeBar,

                    Period = data.Period

                };

            }

            else

            {

                //Aggregate the working bar

                workingBar.Close = data.Close;

                workingBar.Volume += data.Volume;

                workingBar.Period += data.Period;

                if (data.Low < workingBar.Low) workingBar.Low = data.Low;

                if (data.High > workingBar.High) workingBar.High = data.High;

            }

        }

    }





    /// 

    /// EURUSD is a custom data type we create for this algorithm

    /// 


    public class EURUSD : BaseData

    {

        /// 

        /// Opening Price

        /// 


        public decimal Open = 0;

        /// 

        /// High Price

        /// 


        public decimal High = 0;

        /// 

        /// Low Price

        /// 


        public decimal Low = 0;

        /// 

        /// Closing Price

        /// 


        public decimal Close = 0;

       

        /// 

        /// Default initializer for EURUSD.

        /// 


        public EURUSD()

        {

            Symbol = "EURUSD";

        }



        /// 

        /// Return the URL string source of the file. This will be converted to a stream 

        /// 


        public override string GetSource(SubscriptionDataConfig config, DateTime date, DataFeedEndpoint datafeed)

        {

            return @"..\..\..\Data\custom\EURUSD_1M_20100401_20141111.csv";

        }



        /// 

        /// Reader converts each line of the data source into BaseData objects. Each data type creates its own factory method, and returns a new instance of the object 

        /// each time it is called. 

        /// 


        public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, DataFeedEndpoint datafeed)

        {

           

            EURUSD index = new EURUSD();



            try

            {

                string[] data = line.Split(',');

                string timeValue = (Convert.ToInt32(data[2])).ToString("00:00:00");

                string dateValue = (Convert.ToInt32(data[1])).ToString("0000/00/00");

                index.Symbol = data[0];

                index.Time = DateTime.Parse(dateValue + " " + timeValue);

                index.Open = Convert.ToDecimal(data[3], CultureInfo.InvariantCulture);

                index.High = Convert.ToDecimal(data[4], CultureInfo.InvariantCulture);

                index.Low = Convert.ToDecimal(data[5], CultureInfo.InvariantCulture);

                index.Close = Convert.ToDecimal(data[6], CultureInfo.InvariantCulture);

                index.Value = index.Close;

            }

            catch (Exception ex)

            {

                ex.ToString();

            }



            return index;

        }



    }

}

Michael Handschuh

50.5k ,

Hey Greg, in the future please use the 'Attach Project' and 'Attach Backtest Results' -- this makes it much easier for me to clone and help you :) and also helps keep the forum thread looking cleaner, making it easier for others to scan and find solutions to similar problems.

As for the issue you're facing, this stems from the usage of EURUSD type. The DailyTradeBarConsolidator is execting a 'TradeBar' type. I'm unable to prove this works since I don't have your custom data, but please check out the attached project. What I've done is made EURUSD subclass TradeBar instead of BaseData and then I've removed the OHLC fields. This makes the EURUSD type a TradeBar type as well, so the consolidator can work with it. Another option (less awesome) is to modify the consolidator to work on EURUSD data.

This 'should' work! Let me know if you need further help getting this running!

Greg L

444 ,

Thank Michael,

I will use the Attach option in the future. I now see that TradeBar has BaseData as a base class so it was fairly straight forward to covert the EURUSD from BaseData to TradeBar - I should have noticed this sooner. Doing it as you did also keeps the ResolveConsolidator working too.

Thanks again for the help,
Greg

JP B

22.2k ,

@MichaelH Just wondering: why don't you guys include DailyTradeBarConsolidator() in a new release of LEAN?

Also, when backtesting with normal equity data, can I just use ResolveConsolidator("MSFT", Resolution.Daily); or is it better to use the custom DailyTradeBarConsolidator() ?

Michael Handschuh

50.5k ,

The DailyTradeBarConsolidator was put in that code because of a bug in determining trade bar start times. This has since been fixed in the open source project, so the following is correct:var consolidator = new TradeBarConsolidator(TimeSpan.FromDays(1));That being said, using the following is always safe:var consolidator = ResolveConsolidator("SPY", Resolution.Daily);

JP B

22.2k ,

@MichaelH Awesome, thanks for informing.

Greg L INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

Community Ambassadors

Earn free QuantConnect cloud credits by helping others, each week we airdrop credits to the most active community members.

233,163 Quants.

Online Now

Stay Connected