Algorithm took longer than 10 minutes

Here’s an in-depth answer for Dharmesh, including advice and an illustrative code sketch:

Yes, batching your daily bar processing into several scheduled events—each handling a distinct slice of your universe—is a well-established QuantConnect best practice for avoiding single-time-loop 10-minute runtime errors.

Why This Works

• Each scheduled event gets its own 10-minute compute window. By spreading work across three events, you (effectively) triple your available time budget, letting each event process just a fraction of the universe comfortably.
• QuantConnect schedules each callback independently: Your process_daily_bars_1, 2, 3 will not add up their processing times—they have their own limits.

How to Implement Correctly

• Defining Clean Batches:

  • Avoid overlapping or missing symbols in your slices.
  • Use a stable ordering (like a sorted symbol list) if your dictionary order might change.
  • Compute batch indices dynamically based on actual symbol count to handle varying universe size.

• Example:

  • For ~700 symbols, with 3 batches, ceil-divide to get ~234 per batch.
  • For N symbols and K batches, batch_size = ceil(N / K). Each batch operates on its [start:end] index range.

• Consistent State Checks:

  • As you already do: always check for self.IsWarmingUp and daily_bars non-emptiness in each scheduled handler.

• Diagnostics:

  • Log start/end time (use self.Debug) and batch size for each scheduled batch to monitor and tune performance.

• If you still encounter slowness:

  • You can further reduce batch size, increase the number of batches, or—if absolutely necessary—leverage QuantConnect’s Training sessions, which allocate longer run windows.

Clean Code Sketch

Below is a pattern showing how to split your universe into 3 non-overlapping batches, each scheduled as a separate after-market-close event.

from itertools import islice

class MyAlgorithm(QCAlgorithm):

    def initialize(self):
        # ... your normal universe setup ...
        self._batch_count = 3  # Number of batches
        self.schedule.on(self.date_rules.every_day(), self.time_rules.after_market_close("SPY", 1), self.process_daily_bars_1)
        self.schedule.on(self.date_rules.every_day(), self.time_rules.after_market_close("SPY", 11), self.process_daily_bars_2)
        self.schedule.on(self.date_rules.every_day(), self.time_rules.after_market_close("SPY", 21), self.process_daily_bars_3)

    def _get_batch(self, batch_num):
        # Split symbols into batch_num-sized chunks
        symbols = list(self.daily_bars.keys())
        n = len(symbols)
        batch_size = (n + self._batch_count - 1) // self._batch_count  # ceiling division
        start = batch_num * batch_size
        end = min(start + batch_size, n)
        return {s: self.daily_bars[s] for s in symbols[start:end]}

    def process_daily_bars_1(self):
        if self.is_warming_up or not self.daily_bars: return
        batch = self._get_batch(0)
        self._process_slice(batch, 1)

    def process_daily_bars_2(self):
        if self.is_warming_up or not self.daily_bars: return
        batch = self._get_batch(1)
        self._process_slice(batch, 2)

    def process_daily_bars_3(self):
        if self.is_warming_up or not self.daily_bars: return
        batch = self._get_batch(2)
        self._process_slice(batch, 3)

    def _process_slice(self, batch, batch_idx):
        # Your intensive symbol processing here
        for symbol, bar in batch.items():
            self.debug(f"Processed {symbol} in batch {batch_idx}")
        self.debug(f"Batch {batch_idx} processed {len(batch)} symbols at {self.time}")

Additional Tips

• If your universe grows much beyond ~1,000, look into parallel processing (if possible within QuantConnect’s architecture), or pre-filter/summarize input before this stage.
• Only use QuantConnect’s Training if batches simply cannot complete within 10 minutes despite all efforts—your current approach should be sufficient for ~700 if the per-symbol logic is efficient.

In summary:
Your batching/scheduling strategy is the right direction. Just use dynamic batch boundaries and log each batch’s timing for monitoring. You’re on the right path!

Sources: Algorithm Engine - QuantConnect.com, Event Handlers - QuantConnect.com, Opening Range Breakout for Stocks in Play, Scheduled Events - QuantConnect.com, Key Concepts - QuantConnect.com

what is the QuantConnect’s Training ?