Back

Consolidator for custom data

Hi,

I have a custom data type extending BaseData, this data is minute by minute and I'd like to consolidate the bars into hourly bars. I've been looking at the examples but I have not been able to figure this out just yet.

Does anyone have a sample of something similar?

Also fwiw I am running the Lean engine via VS2013 Community Ed. on my Windows 8.1 workstation and the custom data is a csv on the workstation.

Thanks,
Greg L.
Update Backtest








Hey Greg!

Can you post your custom type? I typically just use the built in resolver:
var consolidator = ResolveConsolidator("CustomSymbol", Resolution.Hour);
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Michael,

Thanks for the help, I didn't know to use the ResolveConsolidator - that works great for the hourly bars. Should


var consolidator = ResolveConsolidator("CustomSymbol", Resolution.Daily );


work the same? From my test it did not seem to. When its set to Resolution.Hour the HourBarHandler event is triggered and the Console.Writeline happens.

If I change Resolution.Hour to Resolution.Daily the event does not seem to fire.

Here is my code:

public class TestingCustomDataAlgo : QCAlgorithm
{

///
/// Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.
///

public override void Initialize()
{

SetStartDate(new DateTime(2010, 04, 02, 00, 00, 00));
SetEndDate(2010, 05, 04);

AddData("EURUSD");

//wire up an event to handle the bars
var consolidator = ResolveConsolidator("EURUSD", Resolution.Hour);
//var consolidator = ResolveConsolidator("EURUSD", Resolution.Daily);
consolidator.DataConsolidated += HourBarHandler;
SubscriptionManager.AddConsolidator("EURUSD", consolidator);

}

///
/// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
///

/// TradeBars IDictionary object with your stock data
public void OnData(EURUSD data)
{
//Console.WriteLine("Time: " + data.Time);
}

private void HourBarHandler(object sender, BaseData consolidated)
{
Console.WriteLine("Consolidated Time: " + consolidated.Time);
}

}

///
/// EURUSD is a custom data type we create for this algorithm
///

public class EURUSD : BaseData
{
///
/// Opening Price
///

public decimal Open = 0;
///
/// High Price
///

public decimal High = 0;
///
/// Low Price
///

public decimal Low = 0;
///
/// Closing Price
///

public decimal Close = 0;

///
/// Default initializer for EURUSD.
///

public EURUSD()
{
Symbol = "EURUSD";
}

///
/// Return the URL string source of the file. This will be converted to a stream
///

public override string GetSource(SubscriptionDataConfig config, DateTime date, DataFeedEndpoint datafeed)
{
return @"..\..\..\Data\custom\EURUSD_1M_20100401_20141111.csv";
}

///
/// Reader converts each line of the data source into BaseData objects. Each data type creates its own factory method, and returns a new instance of the object
/// each time it is called.
///

public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, DataFeedEndpoint datafeed)
{

EURUSD index = new EURUSD();

try
{
string[] data = line.Split(',');
string timeValue = (Convert.ToInt32(data[2])).ToString("00:00:00");
string dateValue = (Convert.ToInt32(data[1])).ToString("0000/00/00");
index.Symbol = data[0];
index.Time = DateTime.Parse(dateValue + " " + timeValue);
index.Open = Convert.ToDecimal(data[3], CultureInfo.InvariantCulture);
index.High = Convert.ToDecimal(data[4], CultureInfo.InvariantCulture);
index.Low = Convert.ToDecimal(data[5], CultureInfo.InvariantCulture);
index.Close = Convert.ToDecimal(data[6], CultureInfo.InvariantCulture);
index.Value = index.Close;
}
catch (Exception ex)
{
ex.ToString();
}

return index;
}

}


Basically instead of re-formatting / packaging my csv data I found it easier to manipulate the time and date fields using the custom type Reader method.

Thanks again for the help,
Greg
0

Hey Greg,

I've attached an example running against custom bitcoin data that uses the daily consolidation. This seems to work as expected. I'm unable to run your exact configuration since it looks like you're running LEAN locally against some custom data. If you'd like, put your custom data in dropbox and we can run it here on the cloud to further diagnose your issue.
1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Michael,

Thanks for the help, I'll tinker with it for a bit and see if I can figure out - if that fails I'll do a dropbox link and let you know.

Appreciate your time!
Greg
0

Hi Michael, The daily resolution does work - I just put my test code in the wrong method. I am running into another issue that's related to the data itself, the fact that its forex data to be precise. Because the data contains an abbreviated Sunday session the daily bar ends for this at 21:15 and what I am finding is that this seems to roll forward into all the daily bars after the first Sunday. This is a sample from my testing:

Consolidated Time: Wednesday at 5/5/2010 12:00:00 AM
Consolidated Time: Thursday at 5/6/2010 12:00:00 AM
Consolidated Time: Friday at 5/7/2010 12:00:00 AM
Consolidated Time: Sunday at 5/9/2010 9:15:00 PM
Consolidated Time: Monday at 5/10/2010 9:15:00 PM

Would you have any thoughts/ideas on how I could get the days following a Sunday to start using 00:00 (aka 12:00AM)?

Thanks again for your time,
Greg
0

Hey Greg, I'll submit a bug report to the LEAN github. This is certainly not the desired behavior :).

I've attached a project that defines a DailyTradeBarConsolidator. This should behave closer to as expected:
2015-03-02 00:01:00 Consolidated Time: Sunday - 03/01/2015 19:30:00
2015-03-03 00:01:00 Consolidated Time: Monday - 03/02/2015 00:00:00
2015-03-04 00:01:00 Consolidated Time: Tuesday - 03/03/2015 00:00:00
2015-03-05 00:01:00 Consolidated Time: Wednesday - 03/04/2015 00:00:00
2015-03-06 00:01:00 Consolidated Time: Thursday - 03/05/2015 00:00:00
2015-03-08 17:01:00 Consolidated Time: Friday - 03/06/2015 00:00:00
2015-03-09 00:01:00 Consolidated Time: Sunday - 03/08/2015 17:00:00
2015-03-10 00:01:00 Consolidated Time: Monday - 03/09/2015 00:00:00
2015-03-11 00:01:00 Consolidated Time: Tuesday - 03/10/2015 00:00:00
2015-03-12 00:01:00 Consolidated Time: Wednesday - 03/11/2015 00:00:00
2015-03-13 00:01:00 Consolidated Time: Thursday - 03/12/2015 00:00:00
2015-03-15 17:01:00 Consolidated Time: Friday - 03/13/2015 00:00:00
2015-03-16 00:01:00 Consolidated Time: Sunday - 03/15/2015 17:00:00
2015-03-17 00:01:00 Consolidated Time: Monday - 03/16/2015 00:00:00

I'll work on implementing a more permanent fix to this later this week. Let me know if this fits your needs!
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks Michael, I looked at this briefly and it seems like it will suit my needs just fine. Also thanks for submitting the issue with the LEAN development team.

Thanks again for your help!
Greg
0

Hi Michael,

When I use the DailyTradeBarConsolidator this way


var dtbConsolidator = new DailyTradeBarConsolidator();
dtbConsolidator.DataConsolidated += DayBarHandler;
SubscriptionManager.AddConsolidator("EURUSD", dtbConsolidator);


I get a runtime error like this:

Type mismatch found between consolidator and symbol. Symbol: EURUSD expects type EURUSD but tried to register consolidator with input type TradeBar

Conceptually I think I get difference between a 'symbol' and a 'bar' of that symbol but without actually reading all the source code I'm not seeing how to get a 'TradeBar' from 'BaseData.' Is there a method I can override to return a TradeBar from my type that extends BaseData or do I need to use a different approach?

Thanks again for your assistance,
Greg
0

Please attach the entire project, or a subset of the project showcasing the error. This looks like you're trying to use a custom data type called 'EURUSD' but the consolidator is expecting 'TradeBar' data.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks again for your help! Your understanding is correct and that was my question to which I did not know the answer. Code is below:


using System;
using System.IO;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using QuantConnect.Data;
using QuantConnect.Data.Market;
using QuantConnect.Data.Consolidators;

namespace QuantConnect.Algorithm.Examples
{

///
/// Testing algorithm simply initializes the date range and cash
///

public class TestingCustomDataAlgo : QCAlgorithm
{

///
/// Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.
///

public override void Initialize()
{

SetStartDate(new DateTime(2010, 05, 03, 00, 00, 00));
SetEndDate(2010, 06, 01);

AddData("EURUSD");

//create an event to handle the hourly bars
var consolidator = ResolveConsolidator("EURUSD", Resolution.Daily);
consolidator.DataConsolidated += HourBarHandler;
SubscriptionManager.AddConsolidator("EURUSD", consolidator);

var dtbConsolidator = new DailyTradeBarConsolidator();
dtbConsolidator.DataConsolidated += DayBarHandler;
SubscriptionManager.AddConsolidator("EURUSD", dtbConsolidator);

}

private void DayBarHandler(object sender, BaseData consolidated)
{
Console.WriteLine("\nDBH Consolidated Time: " + consolidated.Time.DayOfWeek + " at " + consolidated.Time);
}

///
/// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
///

/// TradeBars IDictionary object with your stock data
public void OnData(EURUSD data)
{
//Console.WriteLine("Time: " + data.Time);
}

private void HourBarHandler(object sender, BaseData consolidated)
{
Console.WriteLine("\nConsolidated Time: " + consolidated.Time.DayOfWeek + " at " + consolidated.Time);

//Debug("\nConsolidated Daily Time: " + consolidated.Time);
}
}

///
/// ad hoc fix for daily close issue
///

public class DailyTradeBarConsolidator : DataConsolidator
{
private TradeBar _working;

///
/// OutputType returns a Type?
///

public override Type OutputType
{
get { return typeof(TradeBar); }
}

///
/// Update method
///

/// Tradebar type
public override void Update(TradeBar data)
{
if (_working != null && _working.Time.Date != data.Time.Date)
{
OnDataConsolidated(_working);
_working = null;
}

AggregateBar(ref _working, data);
}

///
/// Aggregates the new 'data' into the 'workingBar'. The 'workingBar' will be
/// null following the event firing
///

/// The bar we're building, null if the event was just fired and we're starting a new trade bar
/// The new data
protected void AggregateBar(ref TradeBar workingBar, TradeBar data)
{
if (workingBar == null)
{
workingBar = new TradeBar
{
Time = data.Time,
Symbol = data.Symbol,
Open = data.Open,
High = data.High,
Low = data.Low,
Close = data.Close,
Volume = data.Volume,
DataType = MarketDataType.TradeBar,
Period = data.Period
};
}
else
{
//Aggregate the working bar
workingBar.Close = data.Close;
workingBar.Volume += data.Volume;
workingBar.Period += data.Period;
if (data.Low < workingBar.Low) workingBar.Low = data.Low;
if (data.High > workingBar.High) workingBar.High = data.High;
}
}
}


///
/// EURUSD is a custom data type we create for this algorithm
///

public class EURUSD : BaseData
{
///
/// Opening Price
///

public decimal Open = 0;
///
/// High Price
///

public decimal High = 0;
///
/// Low Price
///

public decimal Low = 0;
///
/// Closing Price
///

public decimal Close = 0;

///
/// Default initializer for EURUSD.
///

public EURUSD()
{
Symbol = "EURUSD";
}

///
/// Return the URL string source of the file. This will be converted to a stream
///

public override string GetSource(SubscriptionDataConfig config, DateTime date, DataFeedEndpoint datafeed)
{
return @"..\..\..\Data\custom\EURUSD_1M_20100401_20141111.csv";
}

///
/// Reader converts each line of the data source into BaseData objects. Each data type creates its own factory method, and returns a new instance of the object
/// each time it is called.
///

public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, DataFeedEndpoint datafeed)
{

EURUSD index = new EURUSD();

try
{
string[] data = line.Split(',');
string timeValue = (Convert.ToInt32(data[2])).ToString("00:00:00");
string dateValue = (Convert.ToInt32(data[1])).ToString("0000/00/00");
index.Symbol = data[0];
index.Time = DateTime.Parse(dateValue + " " + timeValue);
index.Open = Convert.ToDecimal(data[3], CultureInfo.InvariantCulture);
index.High = Convert.ToDecimal(data[4], CultureInfo.InvariantCulture);
index.Low = Convert.ToDecimal(data[5], CultureInfo.InvariantCulture);
index.Close = Convert.ToDecimal(data[6], CultureInfo.InvariantCulture);
index.Value = index.Close;
}
catch (Exception ex)
{
ex.ToString();
}

return index;
}

}
}

0

Hey Greg, in the future please use the 'Attach Project' and 'Attach Backtest Results' -- this makes it much easier for me to clone and help you :) and also helps keep the forum thread looking cleaner, making it easier for others to scan and find solutions to similar problems.

As for the issue you're facing, this stems from the usage of EURUSD type. The DailyTradeBarConsolidator is execting a 'TradeBar' type. I'm unable to prove this works since I don't have your custom data, but please check out the attached project. What I've done is made EURUSD subclass TradeBar instead of BaseData and then I've removed the OHLC fields. This makes the EURUSD type a TradeBar type as well, so the consolidator can work with it. Another option (less awesome) is to modify the consolidator to work on EURUSD data.

This 'should' work! Let me know if you need further help getting this running!
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thank Michael,

I will use the Attach option in the future. I now see that TradeBar has BaseData as a base class so it was fairly straight forward to covert the EURUSD from BaseData to TradeBar - I should have noticed this sooner. Doing it as you did also keeps the ResolveConsolidator working too.

Thanks again for the help,
Greg
1

@MichaelH Just wondering: why don't you guys include DailyTradeBarConsolidator() in a new release of LEAN?

Also, when backtesting with normal equity data, can I just use ResolveConsolidator("MSFT", Resolution.Daily); or is it better to use the custom DailyTradeBarConsolidator() ?
0

The DailyTradeBarConsolidator was put in that code because of a bug in determining trade bar start times. This has since been fixed in the open source project, so the following is correct:var consolidator = new TradeBarConsolidator(TimeSpan.FromDays(1));That being said, using the following is always safe:var consolidator = ResolveConsolidator("SPY", Resolution.Daily);
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


@MichaelH Awesome, thanks for informing.
0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed