# Comprehensive Review & Improvement Request - NewsScanner V7.3.5

---

## 📋 Context:

I'm developing a **NewsScanner** system on QuantConnect to monitor news for Micro-cap/Low Float stocks and send Telegram alerts when volume or price spikes occur.

The system is now working after fixing several errors, but I need a comprehensive review to ensure:
1. ✅ The fixes are correct and appropriate
2. ✅ No hidden bugs remain
3. ✅ The system works as expected
4. ✅ Add a separate Telegram logging bot feature

---

## 🔴 Errors Encountered and Fixes Applied:

### Error 1: Variable Name Conflict (V7.3.1)

**Error Message:**
```python
AttributeError: 'Config' object has no attribute 'MAX_FREE_FLOAT_SHARES'
```

**Root Cause:**
- In `config.py`: Variable defined as `MAX_SHARES_OUTSTANDING = 30_000_000`
- In `telegram_notifier.py`: Code uses `config.MAX_FREE_FLOAT_SHARES` (doesn't exist!)

**Fix Applied:**
```python
# In telegram_notifier.py (line 227):
# Before:
if shares_outstanding > config.MAX_FREE_FLOAT_SHARES:

# After:
if shares_outstanding > config.MAX_SHARES_OUTSTANDING:
```

**Question:** Is this fix correct? Is there a better way to prevent these issues in the future?

---

### Error 2: Runtime Error - format strings with None (V7.3.2)

**Error Message:**
```python
Runtime Error: unsupported format string passed to NoneType.__format__
at fine_selection
   self.log(f"[FineSelection] {x.symbol.value}: Shares={shares_outstanding:,}, MarketCap=${market_cap:,.0f}")
in main.py: line 294
```

**Root Cause:**
When `shares_outstanding` or `market_cap` is `None`, attempting to format with `:,` or `:.0f` causes an error.

**Fix Applied:**
```python
# In main.py (lines 294-296):
# Before:
self.log(f"Shares={shares_outstanding:,}, MarketCap=${market_cap:,.0f}")

# After:
shares_str = f"{shares_outstanding:,}" if shares_outstanding else "N/A"
market_cap_str = f"${market_cap:,.0f}" if market_cap else "$N/A"
self.log(f"Shares={shares_str}, MarketCap={market_cap_str}")
```

**Question:** Is this fix sufficient? Should similar checks be added elsewhere?

---

### Error 3: Shares=N/A - Failed to Retrieve Data (V7.3.3)

**Problem:**
```
[FineSelection] AA: Shares=N/A, MarketCap=$9,017,999,802
[FineSelection] After Market Cap + Shares filter: 0 stocks
```

All stocks showed `Shares=N/A`, meaning data retrieval failed.

**Root Cause:**
Code was using **Python naming convention** (snake_case):
```python
fine_fundamental.company_reference.shares_outstanding  # ❌
x.market_cap  # ❌
```

While QuantConnect uses **C# naming convention** (PascalCase):
```python
fine_fundamental.CompanyReference.SharesOutstanding  # ✅
x.MarketCap  # ✅
```

**Fix Applied:**
```python
# In main.py (lines 328-352):
def _get_shares_outstanding(self, fine_fundamental):
   try:
       # V7.3.3: Use PascalCase (QuantConnect C# naming)
       # Attempt 1: CompanyReference.SharesOutstanding
       if hasattr(fine_fundamental, 'CompanyReference') and hasattr(fine_fundamental.CompanyReference, 'SharesOutstanding'):
           shares = fine_fundamental.CompanyReference.SharesOutstanding
           if shares and shares > 0:
               return shares
       
       # Attempt 2: EarningReports.BasicAverageShares
       if hasattr(fine_fundamental, 'EarningReports') and hasattr(fine_fundamental.EarningReports, 'BasicAverageShares'):
           shares = fine_fundamental.EarningReports.BasicAverageShares.ThreeMonths
           if shares and shares > 0:
               return shares
       
       # Attempt 3: FinancialStatements.SharesOutstanding
       if hasattr(fine_fundamental, 'FinancialStatements') and hasattr(fine_fundamental.FinancialStatements, 'SharesOutstanding'):
           shares = fine_fundamental.FinancialStatements.SharesOutstanding.ThreeMonths
           if shares and shares > 0:
               return shares
       
       return None
   except Exception as e:
       return None
```

**Question:** Is this fix comprehensive? Are there other data sources I should try?

---

### Error 4: Consolidator Error - Resolution Conflict (V7.3.4)

**Error Message:**
```
Live Handled Error: [OnSecuritiesChanged] Error: Unable to create AAME consolidator 
because AAME is registered for Daily data. 
Consolidators require higher resolution data to produce lower resolution data.
```

**Root Cause:**
System was trying to create technical indicators (RSI, MACD, Bollinger Bands) with **Minute resolution** on stocks registered with **Daily resolution**.

**Fix Applied:**
1. In `config.py`: Disable technical indicators
```python
ENABLE_TECHNICAL_INDICATORS = False  # Disabled: requires Minute data
```

2. In `symbol_data.py`: Add check before creating indicators
```python
if config.ENABLE_TECHNICAL_INDICATORS:
   self.technical_indicators = TechnicalIndicators(algorithm, symbol)
else:
   self.technical_indicators = None
```

3. In `telegram_notifier.py`: Add None check
```python
if config.INCLUDE_TECHNICAL_IN_ALERTS and hasattr(symbol_data, 'technical_indicators') and symbol_data.technical_indicators is not None:
   # Use indicators
```

**Questions:** 
- Is disabling technical indicators the optimal solution? 
- Should I change stock resolution to Minute instead?
- What are the costs/benefits of each option?

---

### Error 5: Timezone Error - offset-naive/aware (V7.3.5)

**Error Message:**
```python
Runtime Error: can't subtract offset-naive and offset-aware datetimes
at process_news
   news_age = (self.algo.utc_time - news_item.time).total_seconds()
in news_analyzer.py: line 140
```

**Root Cause:**
Attempting to subtract two datetimes with different timezone awareness:
- `self.algo.utc_time` might be offset-naive
- `news_item.time` might be offset-aware

**Fix Applied:**
```python
# In news_analyzer.py (lines 138-158):
if hasattr(news_item, 'time') and news_item.time:
   try:
       algo_time = self.algo.utc_time
       news_time = news_item.time
       
       # Ensure both datetimes have the same timezone
       if algo_time.tzinfo is None and news_time.tzinfo is not None:
           algo_time = algo_time.replace(tzinfo=pytz.UTC)
       elif algo_time.tzinfo is not None and news_time.tzinfo is None:
           news_time = news_time.replace(tzinfo=pytz.UTC)
       
       news_age = (algo_time - news_time).total_seconds()
       if news_age > 120:  # Older than 2 minutes
           old_news_filtered += 1
           continue
   except Exception as e:
       # In case of any error, treat news as new
       self.algo.debug(f"[NewsAnalyzer] Error calculating news age: {e}")
       pass
```

**Questions:** 
- Is this fix correct for timezone handling?
- Should I use `pytz.utc.localize()` instead of `replace()`?
- Is there a better way to handle timezones in QuantConnect?

---

## 📊 Current Status:

### ✅ What's Working Now:
1. System selects 585 Low Float / Micro-cap stocks
2. Benzinga news feed added for each stock
3. OnData is working and receiving data
4. Verified 6 stocks - all match criteria 100%

### ❓ What Hasn't Been Tested Yet:
1. Are news items being received correctly?
2. Are monitoring windows being opened?
3. Is reaction checking working?
4. Are Telegram alerts being sent?

---

## 🎯 Main Questions:

### 1. Review of Fixes:
**Are the fixes applied for the five errors above correct and appropriate? Are there better approaches?**

### 2. Add Separate Logging Bot:
**I want to add a second Telegram bot (separate from the main alerts bot) to receive all logs:**
- Every `self.log()` message sent to this bot
- For real-time system monitoring
- For quick troubleshooting

**How do I implement this? Where should I add the code?**

### 3. Comprehensive Test Scenario:
**How can I ensure the system works correctly?**

I'm thinking of a test scenario:
1. Generate a fake news item (or wait for real news)
2. Track what happens step by step
3. Verify:
  - ✅ News received
  - ✅ Monitoring window opened
  - ✅ Reaction checked
  - ✅ Alert sent (if conditions met)

**Is there a better testing approach? Can I simulate news in QuantConnect?**

### 4. Success Indicators:
**What metrics indicate the system is working correctly?**

For example:
- Expected number of news items per day?
- Percentage of monitoring windows opened?
- Percentage of alerts sent?

### 5. Other Potential Errors:
**Based on the previous errors, what other potential issues should I watch out for?**

For example:
- Other timezone issues?
- None values in other places?
- Naming convention issues?
- Telegram rate limiting?
- Memory leaks?

### 6. Best Practices:
**What are the best practices for:**
- Error handling in QuantConnect?
- Logging in live trading?
- Handling missing data (None values)?
- Dealing with timezones?
- Testing before live deployment?

---

## 📁 Main Files:

I have 10 Python files:
1. `main.py` - Main file (1082 lines)
2. `config.py` - Configuration (233 lines)
3. `news_analyzer.py` - News analysis (302 lines)
4. `telegram_notifier.py` - Alerts (528 lines)
5. `symbol_data.py` - Stock data management (368 lines)
6. `technical_indicators.py` - Technical indicators (358 lines)
7. `anomaly_detector.py` - Anomaly detection
8. `helpers.py` - Helper functions
9. `translation_service.py` - Translation
10. `update_universe_900_weekly.py` - Universe update

---

## 🎯 Final Goal:

A stable and reliable system that:
1. ✅ Runs 24/7 without errors
2. ✅ Receives and analyzes news correctly
3. ✅ Sends accurate Telegram alerts
4. ✅ Can be monitored and diagnosed easily
5. ✅ Handles errors safely

---

## 📝 Summary of Questions:

1. **Are the five fixes correct? Are there better approaches?**
2. **How do I add a separate Telegram logging bot to receive all logs?**
3. **What's the best scenario to comprehensively test the system?**
4. **What metrics indicate system success?**
5. **What other potential errors should I watch out for?**
6. **What are best practices for error handling, logging, and testing in QuantConnect?**

---

**Thank you in advance for your help!** 🙏