Consolidate daily or weekly bar using extendedMarketHours Minute data

To define a custom weekly method, you can use the CalendarInfo object which contains the start time of the bar in the data time zone and the duration of the consolidation period. Here is an example of how you can define a custom weekly method:

def CustomWeekly(self, dt):
    period = timedelta(days=7)
    timeSpan = timedelta(hours=9, minutes=30)
    newDateTime = dt.date() + timeSpan
    delta = 1 + (int)newDateTime.weekday()
    if delta > 6:
        delta = 0
    start = newDateTime - timedelta(days=delta)
    return CalendarInfo(start, period)

In this method, we first define the period as 7 days. Then we set the start time to 9:30 AM on the first trading day of the week. The weekday() function returns the day of the week as an integer (Monday is 0 and Sunday is 6), so we subtract this from the current date to get the start of the week. We then return a CalendarInfo object with the start time and period.

You can then use this method to create a weekly consolidator:

customWeeklyConsolidator = TradeBarConsolidator(self.CustomWeekly)
customWeeklyConsolidator.DataConsolidated += self.OnWeeklyData
self.SubscriptionManager.AddConsolidator(self.symbol, customWeeklyConsolidator)

This will create weekly bars that start at 9:30 AM on the first trading day of the week and end at 4:00 PM on the last trading day of the week.

Please note that this code is based on the examples provided in the QuantConnect documentation.

And actually, there's another problem when I use minute data to consolidate custom daily data. I noticed that the 3:59 PM close price is not equal to the daily close data. It's slightly different from the daily data you get when you directly add equity with daily resolution.

I understand your concern. The issue you're facing is due to the fact that the market doesn't always close at 4:00 PM, and some weeks have an irregular number of trading days. Unfortunately, the consolidator doesn't have the ability to adjust the end time of the bar based on the actual market close time.

However, you can create a workaround by using the daily consolidator and then aggregating the daily bars into weekly bars in your algorithm. Here's an example of how you can do this:

self.dailyBars = {}

def OnDailyData(self, sender, bar):
    # If it's a new week, create a new bar
    if bar.Time.weekday() == 0:
        self.dailyBars[bar.Symbol] = bar
    # If it's the same week, update the bar
    else:
        self.dailyBars[bar.Symbol].Update(bar)

def OnEndOfDay(self):
    # At the end of each day, check if it's Friday
    if self.Time.weekday() == 4:
        # If it's Friday, emit the weekly bar
        for symbol, bar in self.dailyBars.items():
            self.OnWeeklyData(self, bar)
        # Clear the daily bars
        self.dailyBars.clear()

In this example, OnDailyData is the event handler for the daily consolidator. It creates a new bar at the start of each week (Monday) and updates the bar for the rest of the week. OnEndOfDay is called at the end of each trading day. If it's Friday, it emits the weekly bar and clears the daily bars.

Please note that this is a workaround and may not be perfect. For example, it doesn't handle holidays or early market closes. You may need to adjust it based on your specific needs.

I got it. I can use the consolidated daily data to create the weekly bars.

However, there's another problem when I use minute data to consolidate custom daily data. I noticed that the 3:59 PM close price is not equal to the daily close data. It's slightly different from the daily data you get when you directly add equity with daily resolution.

I've been working on this problem for two days now, and it's quite confusing. It's frustrating to have daily and weekly data available, yet I have to put in so much effort to generate them from minute data...

The difference you're seeing between the 3:59 PM close price from minute data and the daily close price from daily data could be due to the fact that the daily close price is typically the last traded price before the market closes at 4:00 PM, while the 3:59 PM price is the last traded price at or before 3:59 PM.

In live trading, bars have a slight delay, so you may receive them milliseconds after the top of each minute. This means that the 3:59 PM bar could potentially include trades that occurred slightly after 3:59 PM, but before 4:00 PM. This could explain the discrepancy you're seeing.

In backtesting, your algorithm receives data at perfect timing. If you request minute resolution data, your algorithm receives the bars at the top of each minute. However, in live trading, bars have a slight delay, so you may receive them milliseconds after the top of each minute. This could explain the discrepancy you're seeing between the 3:59 PM close price and the daily close price.

If you need the exact daily close price, you might consider using the daily data directly. However, please note that custom data is often timestamped to midnight, but the data point may not be available in reality until several days after that point. If your custom dataset is prone to this delay, your backtest may not fetch the same data at the same time or frequency that your live trading algorithm receives the data, leading to deviations between backtesting and live trading. To avoid issues, ensure the timestamps of your custom dataset are the times when the data points would be available in reality.

For more information, you can refer to the QuantConnect documentation.