Tiingo News Algorithm returns different number of articles when live trading

Here’s a targeted diagnostic and advice checklist for Zachary’s issue of receiving fewer TiingoNews articles in live trading than during backtests, meant for an advanced QuantConnect user:

Key Differences: Live vs. Backtest Data

• Backtest Mode: Historical news is fully formed (“batch” downloaded), with all data available instantly. The QC backtest engine guarantees that all articles up to each simulated timestamp are present, often even those published/crawled near or after the “current” bar.
• Live Mode: You receive news only as it is published or as Tiingo’s crawlers deliver it. QC streams this in close to real-time, subject to minor delays and possible batching by Tiingo. Some articles available in a backtest may not appear live until later—or at all, due to timing and delivery limitations.

Possible API/Data Provider Limitations

• Tiingo Live Feed Throttling: Live endpoints may be rate-limited or subject to batch/buffer delays. Unlike backtesting, you cannot retrieve arbitrary history when running live—only what the provider pushes.
• Coverage Gaps: During non-business hours or slow news periods, fewer articles may actually be published or pushed. QC/Tiingo may also limit duplicate or similar records in live streams to avoid floods.
• Live Historical Window: Even if you poll every 4 hours, Tiingo/QuantConnect only supplies what’s published/pulled in real time, so if a crawl is delayed, you might miss or receive fewer articles compared to historical “replay.”

Concrete Debugging/Logging Steps

1. Log All News Events:

   • In both modes, log arrival timestamp, article headline, symbol, publish date, and crawl date.
   • This helps compare the raw data and timing: are specific articles totally missing, or simply delayed in live mode?

2. Store and Compare Article IDs:

   • Save article identifiers (e.g., URLs/headline+publish date combos) for easy comparison between live and backtests over the same period.

3. Monitor API Exception/Rate Limits:

   • Log for any data-provider errors or rate-limiting events (you may be silently throttled in live mode).

4. Scheduled Logging:

   • Use Scheduled Events to count and print how many articles you receive per hour/day in both modes for side-by-side comparison.

Suggestions to Mitigate and Investigate

• Time Window Buffer: In live, hold news for a longer polling window (e.g., a rolling backlog of 6–8 hours) and filter newly arrived duplicates out, in case of minor live delays.
• Retry/Delayed Article Logic: Optionally, if QC delivers a “late” article, check if your filtering logic hides it by publish date—make sure you’re not excluding delayed but relevant arrivals.
• Direct Tiingo API Test: Outside QC, manually query Tiingo’s live endpoint for the same period and compare with what your live algorithm receives, to rule out QC pipeline delays.
• QC and Tiingo Documentation/Forums: Stay up-to-date with provider announcements or QuantConnect forum posts about temporary integration issues, or noted live-vs-historical discrepancies.

Final Checklist

• Compare detailed logs between live and backtest for headline, symbol, publish, and crawl dates.
• Check for missing, not just delayed, articles.
• Review QC and Tiingo for live-feed limitations or downtime reports.
• Consider logging the number of API calls or poll intervals if you implemented custom polling.

By using these diagnostics—including detailed, timestamped logging both live and in backtest—you’ll directly identify which articles are missing, whether delays or delivery issues are present, and if it’s a provider, QC, or code logic source of the difference. This mirrors methods used by advanced quants to synchronize data receipt realities with research-phase expectations.

Sources: US Equities - QuantConnect.com, Opening Range Breakout for Stocks in Play, Scheduled Events - QuantConnect.com, Data Preparation - QuantConnect.com, Live Analysis - QuantConnect.com