Share New Research 
# Detailed Consultation Request for QuantConnect Expert Bot
## STRATEGY OBJECTIVE
**Primary Goal:** Intraday liquidity monitoring for scalping/day trading (3-minute buy/sell trades)
**Core Concept:**
- Monitor a large universe of stocks (100-300) for accumulation signals
- Use a two-phase batch rotation system to minimize resource usage
- Phase 1: Light monitoring (3 indicators) for all stocks in active batch
- Phase 2: Intensive monitoring (6 indicators) for stocks showing accumulation
- Rotate batches every 5 minutes, but keep promoted stocks active across all batches
---
## CURRENT ARCHITECTURE
### Two-Phase System:
**Phase 1 (Batch Monitoring):**
- Indicators: RDV (Relative Daily Volume), CMF (Chaikin Money Flow), OBV (On Balance Volume)
- Purpose: Scan for accumulation signals
- Scope: 20-100 stocks at a time (batch rotation every 5 minutes)
**Phase 2 (Promoted Stocks):**
- Additional indicators: VWAP, Volume SMA, BOP (Balance of Power)
- Purpose: Detailed monitoring and trading
- Scope: Maximum 10-30 stocks (those showing strong accumulation)
### Key Functions:
1. **`initialize_batch(batch_index)`** - Creates Phase 1 indicators for a new batch of stocks
2. **`create_phase1_indicators(symbol)`** - Creates RDV, CMF, OBV indicators
3. **`are_phase1_indicators_ready(symbol)`** - Checks if all Phase 1 indicators are ready
4. **`rotate_batch()`** - Called every 5 minutes to rotate stocks
5. **`check_phase1_accumulation(symbol, slice)`** - Analyzes Phase 1 signals for promotion
6. **`promote_to_phase2(symbol)`** - Upgrades stock from Phase 1 to Phase 2
7. **`demote_to_phase1(symbol)`** - Downgrades stock from Phase 2 back to Phase 1
8. **`cleanup_phase1_indicators(symbol)`** - Removes and resets Phase 1 indicators
9. **`OnData(slice)`** - Main data handler, processes active batch + promoted stocks
### Current Configuration:
```python
UNIVERSE_SIZE = 100
BATCH_SIZE = 20
ROTATION_INTERVAL_MINUTES = 5
MAX_PROMOTED_STOCKS = 10
RDV_PERIOD = 2
CMF_PERIOD = 3
VWAP_PERIOD = 5
VOLUME_SMA_PERIOD = 10
Resolution.MINUTE # All indicators use minute-resolution data
```
---
## CRITICAL PROBLEM: RDV INDICATOR NEVER BECOMES READY
### Symptoms from Backtest Logs:
```
[IndicatorReady-Diag] AORT | InDict: RDV=True CMF=True OBV=True | Ready: RDV=False CMF=True OBV=True | Samples: RDV=20 CMF=20 OBV=20
[IndicatorReady-Diag] GNK | Ready: RDV=False CMF=True OBV=True | Samples: RDV=11 CMF=11 OBV=11
[IndicatorReady-Diag] BLBD | Ready: RDV=False CMF=True OBV=True | Samples: RDV=23 CMF=23 OBV=23
[Rotation-KEEP] PRAA | Indicators not ready yet, keeping for next rotation
[Rotation-KEEP] CECO | Indicators not ready yet, keeping for next rotation
... (repeats for ALL 100 stocks, batch after batch)
```
**Key Observation:**
- RDV has received 20+ samples but `is_ready = False`
- CMF and OBV become ready immediately with 3-11 samples
- ALL stocks are kept forever (never rotated out) because RDV never becomes ready
- This causes memory buildup and prevents batch rotation from working
### Root Cause Discovery:
After researching QuantConnect documentation, I found:
**RDV = Relative DAILY Volume**
- `period=2` means **2 DAYS of daily data** are required
- We're using `Resolution.MINUTE` which creates a fundamental incompatibility
- RDV expects daily bars, but we're feeding it minute bars
- Even with 100+ minute samples, RDV will never become ready because it needs **2 complete trading days**
**Code creating the issue:**
```python
def create_phase1_indicators(self, symbol):
rdv = self.rdv(symbol, self.RDV_PERIOD, Resolution.MINUTE) # ❌ WRONG
cmf = self.cmf(symbol, self.CMF_PERIOD, Resolution.MINUTE) # ✅ OK
obv = self.obv(symbol, Resolution.MINUTE) # ✅ OK
```
---
## PREVIOUS PROBLEMS & SOLUTIONS ATTEMPTED
### Problem 1: OutOfMemoryException
**Original Approach:** Used `warm_up_indicator()` to pre-load historical data for each indicator
```python
rdv = self.rdv(symbol, self.RDV_PERIOD, Resolution.MINUTE)
self.warm_up_indicator(symbol, rdv, Resolution.MINUTE) # Loads historical data
```
**Issue:**
- Loading historical data for 100 stocks × 3 indicators × multiple periods = massive memory usage
- Resulted in `System.OutOfMemoryException`
**Solution Applied:**
- Removed `warm_up_indicator()` calls
- Let indicators warm up naturally with incoming live data
- Reduced memory usage by ~75%
### Problem 2: Type Conversion Errors
**Error Message:**
```
Runtime Error: Trying to dynamically access a method that does not exist throws a TypeError
exception. To prevent the exception, ensure each parameter type matches those required by
the 'float'>) method.
at create_phase1_indicators
rdv.update(IndicatorDataPoint(time, close_price))
```
**Attempted Solutions:**
1. Manual indicator updates with `IndicatorDataPoint`
2. Manual warmup using `history()` + loop + `indicator.update()`
3. Type conversions: `Extensions.to_decimal()`, `int()`, etc.
**Current Solution:**
- Removed all manual warmup
- Rely on QuantConnect's automatic indicator updates via subscriptions
### Problem 3: Stocks Removed Before Indicators Ready
**Issue:** Original code removed stocks from batch after 5 minutes regardless of indicator readiness
**Solution Applied:**
```python
def rotate_batch(self):
symbols_kept = []
for symbol in self.active_batch_symbols:
if not self.are_phase1_indicators_ready(symbol):
symbols_kept.append(symbol)
self.debug(f"[Rotation-KEEP] {symbol.value} | Indicators not ready yet, keeping for next rotation")
continue
# Only remove if ready
self.cleanup_phase1_indicators(symbol)
```
---
## CURRENT BACKTEST ERRORS
### Error 1: All Stocks Perpetually "Not Ready"
- RDV never becomes ready (as explained above)
- Stocks accumulate indefinitely in active batch
- No rotation actually happens
- Eventually hits memory limits
### Error 2: Rate Limiting
```
Your algorithm messaging has been rate limited to prevent browser flooding.
```
- Too many debug messages from keeping 100 stocks perpetually
- Suggests fundamental design issue
---
## QUESTIONS FOR EXPERT BOT
### 1. **Indicator Selection for Intraday/Scalping**
**Current:** RDV (Relative Daily Volume) with `period=2`, `Resolution.MINUTE`
**Problems:**
- RDV is designed for daily resolution
- Never becomes ready with minute data
- Incompatible with scalping strategy
**Question:**
What are the **best QuantConnect indicators** for detecting accumulation/distribution in **minute-resolution** intraday trading?
**Requirements:**
- Must work reliably with `Resolution.MINUTE`
- Should detect volume/price accumulation patterns
- Should warm up quickly (< 10 minutes preferred)
- Suitable for 3-10 minute holding periods
**Current alternatives I'm considering:**
- Volume SMA ratio (current_volume / SMA_volume)
- VWAP deviation
- Price Rate of Change (ROC)
- Money Flow Index (MFI)
Which would you recommend for Phase 1 (light monitoring)?
### 2. **Indicator Warm-up Strategy**
**Three approaches available:**
**A) `warm_up_indicator()` (original approach):**
```python
rdv = self.rdv(symbol, period, resolution)
self.warm_up_indicator(symbol, rdv, resolution)
```
- Pros: Indicators ready immediately
- Cons: OutOfMemoryException with many stocks
**B) Natural warm-up (current approach):**
```python
rdv = self.rdv(symbol, period, resolution)
self.phase1_rdv_indicators[symbol] = rdv
# Auto-updates with subscription data
```
- Pros: No memory issues
- Cons: Indicators take time to warm up
**C) Manual warm-up (expert recommendation):**
```python
history = self.history(symbol, period, resolution)
for time, row in history.iterrows():
indicator.update(time, row['close'])
```
- Pros: Full control, explicit
- Cons: Complex, type conversion issues
**Question:**
For a batch rotation system processing 20-100 stocks every 5 minutes:
- Which warm-up approach is best?
- How do I handle stocks whose indicators aren't ready when batch rotation occurs?
- Should I increase rotation interval to allow more warm-up time?
- Is there a way to pre-warm indicators efficiently without memory issues?
### 3. **Batch Rotation Logic**
**Current flow:**
1. Every 5 minutes, `rotate_batch()` is called
2. For each stock in active batch:
- If indicators ready: remove from batch
- If indicators NOT ready: keep for next rotation
3. Add new batch of stocks
4. Result: Stocks accumulate because RDV never ready
**Question:**
What is the **best practice** for batch rotation with indicator warm-up?
**Options:**
- Should I force-remove stocks after N rotations even if not ready?
- Should I stagger indicator creation (create some, wait, create more)?
- Should I use a "ready queue" and only promote stocks when fully warmed?
- Should rotation interval be dynamic based on readiness rate?
### 4. **Memory Optimization**
**Current stats:**
- Universe: 100 stocks
- Batch: 20 stocks × 3 indicators = 60 indicators
- Promoted: 10 stocks × 6 indicators = 60 indicators
- Total: ~120 active indicators
**Question:**
Are these reasonable numbers for QuantConnect?
**Follow-ups:**
- What's the practical limit for concurrent indicators?
- Should I reduce batch size further?
- Is there overhead from creating/destroying indicators frequently?
- Should I reuse indicator objects instead of recreating?
### 5. **Indicator Readiness Detection**
**Current approach:**
```python
def are_phase1_indicators_ready(self, symbol):
if symbol not in self.phase1_rdv_indicators:
return False
return all([
self.phase1_rdv_indicators[symbol].is_ready,
self.phase1_cmf_indicators[symbol].is_ready,
self.phase1_obv_indicators[symbol].is_ready
])
```
**Diagnostic output shows:**
```
RDV=False CMF=True OBV=True | Samples: RDV=20 CMF=20 OBV=20
```
**Question:**
- Why would an indicator have 20 samples but `is_ready=False`?
- Is there a warmup_period property I should check?
- Should I use `samples >= period` instead of `is_ready`?
- Are there indicators that never become "ready"?
### 6. **Resolution Mismatch Issues**
**Current setup:**
- Universe resolution: `Resolution.MINUTE`
- Indicator resolution: `Resolution.MINUTE`
- Subscription: `self.add_equity(symbol, Resolution.MINUTE)`
**Question:**
Can I safely mix resolutions? For example:
- Use `Resolution.DAILY` for RDV (if keeping it)
- Use `Resolution.MINUTE` for CMF, OBV
- Will this cause data alignment issues?
- How do I ensure indicators update on the same timeframe?
### 7. **Best Practices for Indicator Lifecycle**
**Current approach:**
```python
def cleanup_phase1_indicators(self, symbol):
rdv = self.phase1_rdv_indicators.pop(symbol, None)
if rdv is not None:
rdv.reset() # Is this correct?
```
**Questions:**
- Should I call `reset()` before removing indicators?
- Should I call any dispose/cleanup methods?
- Can I reuse indicators for the same symbol across rotations?
- How do I ensure no memory leaks from indicator objects?
### 8. **Alternative Architecture Suggestions**
**Current:** Two-phase batch rotation (explained above)
**Question:**
Is there a **better architecture** for this use case?
**Constraints:**
- Must monitor 100-300 stocks
- Must detect accumulation patterns quickly
- Must use minimal resources (no 1800 indicators)
- Must support intraday/scalping (minute resolution)
**Ideas to evaluate:**
- Should I use a single phase with fewer indicators?
- Should I use `consolidators` instead of direct subscriptions?
- Should I pre-filter universe to fewer stocks before monitoring?
- Should I use `scheduled_events` differently?
---
## ADDITIONAL CONTEXT
### Universe Filtering:
```python
def coarse_filter_function(self, coarse):
filtered = [x for x in coarse if
x.has_fundamental_data and
self.MIN_PRICE <= x.price <= self.MAX_PRICE and
x.dollar_volume >= self.MIN_DOLLAR_VOLUME]
return [x.symbol for x in sorted(filtered, key=lambda x: x.dollar_volume, reverse=True)[:self.UNIVERSE_SIZE]]
def fine_filter_function(self, fine):
filtered = [f for f in fine if
self.MIN_SHARES_OUTSTANDING <= f.company_profile.shares_outstanding <= self.MAX_SHARES_OUTSTANDING and
self.MIN_MARKET_CAP <= f.market_cap <= self.MAX_MARKET_CAP]
return [f.symbol for f in filtered]
```
### Promotion Logic:
```python
def check_phase1_accumulation(self, symbol, slice):
rdv = self.phase1_rdv_indicators[symbol].current.value
cmf = self.phase1_cmf_indicators[symbol].current.value
obv = self.phase1_obv_indicators[symbol].current.value
is_rdv_elevated = rdv >= 1.00
is_cmf_positive = cmf > 0
is_obv_rising = obv > previous_obv
signal_count = sum([is_rdv_elevated, is_cmf_positive, is_obv_rising])
if signal_count >= 2:
# Promote to Phase 2
return True
```
### Scheduled Events:
```python
self.schedule.on(
self.date_rules.every_day(),
self.time_rules.every(timedelta(minutes=self.ROTATION_INTERVAL_MINUTES)),
self.rotate_batch
)
```
---
## WHAT I NEED FROM YOU
### 1. **Immediate Fix:**
- Replace RDV with appropriate minute-resolution indicator(s)
- Ensure all Phase 1 indicators work reliably with `Resolution.MINUTE`
- Provide exact indicator creation code
### 2. **Architecture Review:**
- Validate if batch rotation approach is sound
- Suggest improvements to warm-up strategy
- Identify any fundamental design flaws
### 3. **Best Practices:**
- QuantConnect-specific patterns for indicator management
- Memory optimization techniques
- Proper cleanup/disposal methods
### 4. **Implementation Guidance:**
- Step-by-step code for fixing indicator issues
- Recommended periods for each indicator
- Thresholds for promotion/demotion logic
### 5. **Testing Strategy:**
- How to verify indicators are warming up correctly
- What metrics to log for debugging
- How to validate batch rotation is working
---
## CONSTRAINTS & LIMITATIONS
1. **Must be QuantConnect-compatible** - No custom libraries
2. **Must use built-in indicators** - No complex custom indicator classes
3. **Must support backtesting** - No live-only features
4. **Must be memory-efficient** - Already hit OutOfMemory errors
5. **Must work with fundamental universe selection** - Coarse/Fine filters
---
## SAMPLE BACKTEST CONFIGURATION
```python
Start Date: 2024-01-01
End Date: 2024-06-01
Initial Capital: $100,000
Time Zone: America/New_York
Universe: US Equities, $1-$30 price range, liquid stocks
```
---
## EXPECTED OUTPUT FROM YOU
Please provide:
1. **Recommended indicators** for Phase 1 (3 indicators max)
2. **Complete code** for indicator creation and warmup
3. **Validation logic** to ensure indicators are ready
4. **Best practices** for batch rotation timing
5. **Memory optimization** recommendations
6. **Any architectural changes** you suggest
7. **Step-by-step implementation plan**
Please be specific with QuantConnect API calls and include actual code examples, not just concepts.
Thank you!
Mia Alissi
EXPLANATION & STEP-BY-STEP GUIDANCE:
1. Recommended Indicators for MINUTE Resolution
2. Warm-up & Creation
self.register_indicator()to bind each to minute bars.self.warm_up_indicator()for fast, controlled history warm-up (will not balloon memory for small batches).3. Readiness Logic
.is_readyproperty — as soon as all three are.is_ready, the symbol is considered "ready"..is_readyis best-practice QuantConnect style.4. Batch Rotation & Cleanup
.reset()on indicators before deleting them to free memory..pop(),del).5. Memory Optimization
self.debug()for logging readiness/health, but throttle to avoid rate limit.6. Debugging & Testing
7. Architecture
8. No memory leaks
.reset()and ensure reference deletion.This template will get you robust, QuantConnect-compatible, intraday volume/accumulation monitoring with batch rotation.
Sources: Accumulation Distribution - QuantConnect.com, docs writing algorithms indicators automatic indicators, Indicator Suite Algorithm, Key Concepts - QuantConnect.com, docs writing algorithms indicators manual indicators
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
مطلق الشمري
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!