Help mia

# Follow-up Question: KeyError in Manual Indicator Warm-up & Additional Questions

Hello again,

Thank you for the excellent guidance on batch rotation and indicator management. I've implemented your recommendations, but encountered a **KeyError** during manual warm-up. I also have additional questions about best practices.

---

## Problem Encountered: KeyError in Manual Warm-up

### Error Details

```
Runtime Error: Trying to retrieve an element from a collection using a key that does not exist in that collection throws a KeyError exception.

at create_phase1_indicators
self.warm_up_indicator(symbol, self.phase1_obv_indicators[symbol], Resolution.MINUTE)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
in main.py: line 451

at initialize_batch
self.create_phase1_indicators(symbol)
in main.py: line 382

at rotate_batch
self.initialize_batch(self.current_batch_index)
in main.py: line 423
```

### Current Implementation (Following Your Advice)

I implemented manual warm-up as you suggested:

```python
def create_phase1_indicators(self, symbol):
"""Create Phase 1 indicators with manual warm-up (Bot recommendation #5)"""
indicators_created = []

try:
# Check liquidity first (Bot recommendation #3)
history = self.history(symbol, max(self.RDV_PERIOD, self.CMF_PERIOD, 5), Resolution.MINUTE)
if history.empty or len(history) < 2:
self.debug(f"[SKIP-ILLIQUID] {symbol.value} | No sufficient history data")
return False

# Create RDV with separate error handling
try:
rdv = self.rdv(symbol, self.RDV_PERIOD, Resolution.MINUTE)
# Manual warm-up (Bot recommendation #5)
for time, row in history.iterrows():
if 'close' in row and row['close'] > 0:
rdv.update(time, row['close'])
self.phase1_rdv_indicators[symbol] = rdv
indicators_created.append(f"RDV(ready={rdv.is_ready}, samples={rdv.samples})")
except Exception as e:
self.error(f"[ERROR] Failed to create RDV for {symbol.value}: {str(e)}")

# Create CMF with separate error handling
try:
cmf = self.cmf(symbol, self.CMF_PERIOD, Resolution.MINUTE)
# Manual warm-up
for time, row in history.iterrows():
if all(k in row for k in ['close', 'high', 'low', 'volume']):
cmf.update(TradeBar(time, symbol, row['open'], row['high'],
row['low'], row['close'], row['volume']))
self.phase1_cmf_indicators[symbol] = cmf
indicators_created.append(f"CMF(ready={cmf.is_ready}, samples={cmf.samples})")
except Exception as e:
self.error(f"[ERROR] Failed to create CMF for {symbol.value}: {str(e)}")

# Create OBV with separate error handling
try:
obv = self.obv(symbol, Resolution.MINUTE)
# Manual warm-up
for time, row in history.iterrows():
if all(k in row for k in ['close', 'volume']):
obv.update(TradeBar(time, symbol, row['open'], row['high'],
row['low'], row['close'], row['volume']))
self.phase1_obv_indicators[symbol] = obv
indicators_created.append(f"OBV(ready={obv.is_ready}, samples={obv.samples})")
except Exception as e:
self.error(f"[ERROR] Failed to create OBV for {symbol.value}: {str(e)}")

# Log results
if indicators_created:
self.debug(f"[IndicatorCreation-IMPROVED] {symbol.value} | {', '.join(indicators_created)}")
return True
else:
self.debug(f"[SKIP-NO-INDICATORS] {symbol.value} | Failed to create any indicators")
return False

except Exception as e:
self.error(f"[ERROR] General error in create_phase1_indicators for {symbol.value}: {str(e)}")
return False
```

### Questions About the KeyError

1. **Why does the KeyError occur?**
- The error happens at line 451 which seems to be the old `warm_up_indicator()` call
- But I thought I replaced all `warm_up_indicator()` calls with manual warm-up
- Could there be a caching issue or is the code not fully updated?

2. **Is my manual warm-up implementation correct?**
- Am I creating TradeBar objects correctly for CMF and OBV?
- Should I use `TradeBar` or just pass the row directly?
- Do I need to handle missing OHLCV data differently?

3. **Should I check if indicator creation succeeded before adding to dictionary?**
```python
# Current approach:
obv = self.obv(symbol, Resolution.MINUTE)
# ... warm up ...
self.phase1_obv_indicators[symbol] = obv

# Better approach?
obv = self.obv(symbol, Resolution.MINUTE)
if obv is not None: # Check if creation succeeded?
# ... warm up ...
self.phase1_obv_indicators[symbol] = obv
```

4. **What if indicator creation fails silently?**
- Can `self.rdv()`, `self.cmf()`, or `self.obv()` return None?
- Or do they always return an object even if data is insufficient?
- How do I detect if indicator creation actually failed?

---

## Additional Questions

### 1. Indicator Update Methods

**Question:** What is the correct way to update different indicator types manually?

For RDV (price-based):
```python
rdv.update(time, row['close']) # ✅ Works?
```

For CMF and OBV (volume-based):
```python
# Option A: Create TradeBar
cmf.update(TradeBar(time, symbol, open, high, low, close, volume))

# Option B: Pass row directly?
cmf.update(time, row)

# Option C: Use different method?
cmf.update(TradeBar.from_dict(row))
```

Which approach is correct for each indicator type?

---

### 2. Handling Missing Data in History

**Question:** How should I handle incomplete or missing data in history?

Current approach:
```python
history = self.history(symbol, 10, Resolution.MINUTE)
if history.empty or len(history) < 2:
return False # Skip this symbol
```

But what if:
- History has 10 rows but some have zero volume?
- History has gaps (missing minutes)?
- OHLC values are present but invalid (negative, NaN)?

Should I:
```python
# Clean the data first?
history = history[history['volume'] > 0]
history = history[history['close'] > 0]
history = history.dropna()

# Then check if enough data remains?
if len(history) < indicator.warm_up_period:
return False
```

---

### 3. Indicator Readiness After Manual Warm-up

**Question:** After manual warm-up, should indicators be immediately ready?

Expected behavior:
```python
rdv = self.rdv(symbol, 2, Resolution.MINUTE)
# Manually feed 2 bars
rdv.update(time1, close1)
rdv.update(time2, close2)
# Should rdv.is_ready be True now?
```

Or does `is_ready` only work with automatic data feed?

---

### 4. Best Practice for Indicator Creation Flow

**Question:** What is the recommended order of operations?

**Option A: Create → Store → Warm-up**
```python
rdv = self.rdv(symbol, 2, Resolution.MINUTE)
self.phase1_rdv_indicators[symbol] = rdv # Store first
# Then warm up
for time, row in history.iterrows():
self.phase1_rdv_indicators[symbol].update(time, row['close'])
```

**Option B: Create → Warm-up → Store**
```python
rdv = self.rdv(symbol, 2, Resolution.MINUTE)
# Warm up first
for time, row in history.iterrows():
rdv.update(time, row['close'])
# Then store
self.phase1_rdv_indicators[symbol] = rdv
```

**Option C: Create → Warm-up → Verify → Store**
```python
rdv = self.rdv(symbol, 2, Resolution.MINUTE)
# Warm up
for time, row in history.iterrows():
rdv.update(time, row['close'])
# Verify it worked
if rdv.is_ready and rdv.samples >= 2:
self.phase1_rdv_indicators[symbol] = rdv
else:
self.debug(f"RDV warm-up failed for {symbol.value}")
```

Which option is best practice?

---

### 5. Error Recovery Strategy

**Question:** If indicator creation fails for one stock, what should I do?

**Current approach:**
- Try to create all 3 indicators
- If any fail, log error but continue
- Return True if at least one indicator was created

**Is this correct, or should I:**
- Return False if ANY indicator fails (all-or-nothing)?
- Keep partial indicators and mark the stock as "incomplete"?
- Retry indicator creation on next rotation?

---

### 6. Performance Optimization

**Question:** Is there a more efficient way to bulk warm-up indicators?

Current approach (per symbol):
```python
for symbol in batch:
history = self.history(symbol, 10, Resolution.MINUTE)
# Create and warm up indicators
```

Better approach (bulk request)?
```python
# Request history for all symbols at once?
all_history = self.history(batch_symbols, 10, Resolution.MINUTE)
for symbol in batch_symbols:
symbol_history = all_history[symbol]
# Create and warm up indicators
```

Does bulk history request improve performance?

---

### 7. Indicator Memory Management

**Question:** After removing a stock from the batch, how do I properly clean up indicators?

Current approach:
```python
def cleanup_phase1_indicators(self, symbol):
self.phase1_rdv_indicators.pop(symbol, None)
self.phase1_cmf_indicators.pop(symbol, None)
self.phase1_obv_indicators.pop(symbol, None)
```

Should I also:
```python
# Explicitly dispose of the indicator?
if symbol in self.phase1_rdv_indicators:
indicator = self.phase1_rdv_indicators[symbol]
# indicator.dispose() ?
# indicator.reset() ?
del self.phase1_rdv_indicators[symbol]
```

Or is `pop()` sufficient for garbage collection?

---

### 8. Monitoring Warm-up Success Rate

**Question:** How can I track and optimize warm-up success rate?

Should I maintain statistics like:
```python
self.warmup_stats = {
'total_attempts': 0,
'successful': 0,
'failed_no_history': 0,
'failed_indicator_creation': 0,
'partial_success': 0
}
```

And log them periodically:
```python
success_rate = self.warmup_stats['successful'] / self.warmup_stats['total_attempts']
self.log(f"[WARMUP-STATS] Success rate: {success_rate:.1%}")
```

Is this recommended for production strategies?

---

## Summary of Questions

1. Why does KeyError occur if I replaced `warm_up_indicator()`?
2. Is my manual warm-up implementation correct?
3. How to properly update different indicator types?
4. How to handle missing/incomplete data in history?
5. Should indicators be immediately ready after manual warm-up?
6. What is the best order: Create → Store → Warm-up?
7. What error recovery strategy should I use?
8. Is bulk history request more efficient?
9. How to properly clean up indicators?
10. Should I track warm-up success rate?

---

## Context

- **Strategy Type**: Intraday with batch rotation
- **Batch Size**: 100 stocks
- **Rotation Interval**: 15 minutes (increased from 5)
- **Indicators**: RDV (period=2), CMF (period=3), OBV
- **Implementation**: Following your previous recommendations

Thank you for your continued guidance!

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Here are robust, best-practice answers and code examples for your questions, especially how to implement indicator warm-up and management for a batch universe, as well as idiomatic handling of edge conditions, performance, and memory management.

1. Why does KeyError occur if I replaced `warm_up_indicator()`?

Cause: Somewhere in your codebase (possibly an old function or in a place you missed), self.warm_up_indicator(symbol, self.phase1_obv_indicators[symbol], ...) is still being called. If the indicator dictionary doesn’t contain the symbol key at that point, KeyError occurs.
Solution: Search your entire code for any warm_up_indicator usage and remove/replace it, especially in inherited, utility, or batch setup classes/methods.

2-3. Manual Warm-Up – Correctness and Usage

The Correct Approach for Manual Warm-up

Price-only indicators (like RDV):

for t, row in history.iterrows():
    if row['close'] > 0:
        rdv.update(t, row['close'])

OHLCV (bar-based) indicators:

for t, row in history.iterrows():
    # Defensive: check for all fields
    if all(k in row for k in ['open', 'high', 'low', 'close', 'volume']):
        bar = TradeBar(t, symbol, row['open'], row['high'], row['low'], row['close'], row['volume'])
        cmf.update(bar)
        obv.update(bar)

Do NOT pass a row/dict directly – always update bar-based indicators with a TradeBar (or the appropriate QuantConnect data class).

4. Handling Missing/Incomplete Data

Best Practice:

history = history.dropna()
history = history[(history['close'] > 0) & (history['volume'] > 0)]
if len(history) < INDICATOR_PERIOD:
    return False  # Not enough valid data

Clean first, then check that enough data remains for indicator warm-up. This avoids silent indicator failures due to bad bars.

5. After Manual Warm-up Will `.is_ready` be True?

Yes: Provided you have given as many valid bars as the indicator's period (some add 1-2 bars of lead/lag internally), .is_ready will be True after manual update, just as with live data.

6. Best Order: Create → Warm-up → Verify → Store

Recommended (Best Practice):

rdv = self.rdv(symbol, 2, Resolution.MINUTE)
for t, r in history.iterrows():
    rdv.update(t, r['close'])
if rdv.is_ready and rdv.samples >= 2:
    self.phase_1_rdv_indicators[symbol] = rdv
else:
    self.debug(f"RDV warm-up failed for {symbol.value}")

This ensures you don’t store “dead” indicator objects.

7. Error Recovery Strategy

Production Practice: Only add fully verified indicators to your dicts, mark “partial” stocks if you wish, and log failures. You may retry those on the next universe/batch update.
Decide on the logic that fits your strategy: all-or-nothing (strict), partial allowed (lenient).

8. Bulk History Request for Efficiency

Bulk history is much faster:

symlist = [symA, symB, symC, ...]
hist = self.history(symlist, 10, Resolution.MINUTE)
for sym in symlist:
    subhist = hist.loc[sym]
    # ...warmup...

Bulk fetches always outperform repeated one-row calls, especially on QuantConnect Cloud.

9. Proper Indicator Cleanup

ind = self.phase_1_rdv_indicators.pop(symbol, None)
if ind and hasattr(ind, 'reset'):
    ind.reset()
# (dispose() is rarely needed in QC; pop() ensures GC if no refs remain)

For most stock universes, just popping out of your dict is sufficient.

10. Monitoring and Logging Warm-up Success Rate

Recommended for research, debugging, and production:
Track stats on attempts, failures, partial successes, etc, as shown below:

self.warmup_stats = {'total': 0, 'success': 0, 'failed': 0}
self.warmup_stats['total'] += 1
# ...after logic...
self.warmup_stats['success'] += 1
self.log(f"Warmup rate: {self.warmup_stats['success']/self.warmup_stats['total']:.2%}")

This helps spot data/logic issues in batch operations.

Robust Indicator Creation: Full Example

def create_phase_1_indicators(self, symbol):
    success = True
    info = {}

    # --- HISTORY CLEANUP ---
    n_warmup = max(self.rdv_period, self.cmf_period, 5)
    history = self.history(symbol, n_warmup * 3, Resolution.MINUTE)
    history = history.dropna()
    history = history[(history['close'] > 0) & (history['volume'] > 0)]
    if len(history) < n_warmup:
        info['history'] = 'Insufficient clean rows'
        self.debug(f"{symbol}: Not enough clean data.")
        return 'failure', info

    # --- CREATE, WARM-UP, VERIFY, STORE ---
    added = {}

    # RDV
    try:
        rdv = self.rdv(symbol, self.rdv_period, Resolution.MINUTE)
        for t, row in history.iterrows():
            rdv.update(t, row['close'])
        if rdv.is_ready and rdv.samples >= self.rdv_period:
            self.phase_1_rdv_indicators[symbol] = rdv
            added['rdv'] = 'ready'
        else:
            info['rdv'] = f"not ready ({rdv.samples})"
            success = False
    except Exception as e:
        info['rdv'] = str(e)
        success = False

    # CMF
    try:
        cmf = self.cmf(symbol, self.cmf_period, Resolution.MINUTE)
        for t, row in history.iterrows():
            tb = TradeBar(t, symbol, row['open'], row['high'], row['low'], row['close'], row['volume'])
            cmf.update(tb)
        if cmf.is_ready and cmf.samples >= self.cmf_period:
            self.phase_1_cmf_indicators[symbol] = cmf
            added['cmf'] = 'ready'
        else:
            info['cmf'] = f"not ready ({cmf.samples})"
            success = False
    except Exception as e:
        info['cmf'] = str(e)
        success = False

    # OBV
    try:
        obv = self.obv(symbol, Resolution.MINUTE)
        for t, row in history.iterrows():
            tb = TradeBar(t, symbol, row['open'], row['high'], row['low'], row['close'], row['volume'])
            obv.update(tb)
        if obv.is_ready and obv.samples >= 2:
            self.phase_1_obv_indicators[symbol] = obv
            added['obv'] = 'ready'
        else:
            info['obv'] = f"not ready ({obv.samples})"
            success = False
    except Exception as e:
        info['obv'] = str(e)
        success = False

    # Results
    if added:
        self.debug(f"{symbol}: indicators created: {added}")
        return ('success' if success else 'partial'), {**info, **added}
    else:
        self.debug(f"{symbol}: no indicators could be created")
        return 'failure', info

Utility/Helper Examples (all per QuantConnect best practice):

# Bulk history and slicing for batch
def bulk_indicator_warmup(self, symbols, period):
    all_hist = self.history(symbols, period*3, Resolution.MINUTE)
    for sym in symbols:
        sym_hist = all_hist.loc[sym]
        # ... indicator creation

# Indicator memory management
def cleanup_indicators(self, symbol):
    for d in [self.phase_1_rdv_indicators, self.phase_1_cmf_indicators, self.phase_1_obv_indicators]:
        ind = d.pop(symbol, None)
        if ind and hasattr(ind, 'reset'):
            ind.reset()

If you implement the above strategy and code, your batch indicator pipeline will be robust, memory safe, quick for warm-up, and easy to debug or extend!

If you have detailed error messages or encounter issues at a specific step, copy the precise error and I'll help you troubleshoot further.

Sources: Key Concepts - QuantConnect.com, Automatic Indicators - QuantConnect.com, Warm Up Periods - QuantConnect.com, Indicator Warmup Algorithm, Warmup Algorithm

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

TOP 5 Research Publications

444,200 Quants.

VOTE FOR UPCOMING FEATURES

Help mia

1. Why does KeyError occur if I replaced `warm_up_indicator()`?

2-3. Manual Warm-Up – Correctness and Usage

The Correct Approach for Manual Warm-up

4. Handling Missing/Incomplete Data

5. After Manual Warm-up Will `.is_ready` be True?

6. Best Order: Create → Warm-up → Verify → Store

7. Error Recovery Strategy

8. Bulk History Request for Efficiency

9. Proper Indicator Cleanup

10. Monitoring and Logging Warm-up Success Rate

Robust Indicator Creation: Full Example

Utility/Helper Examples (all per QuantConnect best practice):

Allocate to this Strategy

Organization

Team

Clone Strategy

Previous Ranking

IN THIS RESEARCH

PARTICIPANTS

Discussion Awards

Actions

Join QuantConnect for Free

SIGN IN

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

TOP 5 Research Publications

444,200 Quants.

VOTE FOR UPCOMING FEATURES

Help mia

1. Why does KeyError occur if I replaced warm_up_indicator()?

2-3. Manual Warm-Up – Correctness and Usage

The Correct Approach for Manual Warm-up

4. Handling Missing/Incomplete Data

5. After Manual Warm-up Will .is_ready be True?

6. Best Order: Create → Warm-up → Verify → Store

7. Error Recovery Strategy

8. Bulk History Request for Efficiency

9. Proper Indicator Cleanup

10. Monitoring and Logging Warm-up Success Rate

Robust Indicator Creation: Full Example

Utility/Helper Examples (all per QuantConnect best practice):

Allocate to this Strategy

Organization

Team

Clone Strategy

Previous Ranking

IN THIS RESEARCH

PARTICIPANTS

Discussion Awards

SHARE RESEARCH

SHARE DISCUSSION

SHARE ARTICLE

SHARE

Actions

Join QuantConnect for Free

1. Why does KeyError occur if I replaced `warm_up_indicator()`?

5. After Manual Warm-up Will `.is_ready` be True?