Gaps/bad data by Mike G

Mike G

, edit

Hello all.

Is it theoretically possible to get bad or no data at a givent moment at the selected resolution, and how does the QC platform handle these?

An example, if my order entry/exit is based on a certain price level, and then an empty bar arrives, would the tradebar fields come in as zeros or something else? Would that be the time to use "fill forward"?

Is there a status indicator of the tradebar that would show OK, READY or similar? If not, what would be a good way to protect agains submitting orders in the event of bad or missing data?

If consolidated bars are used, for example 10 minute bars, and during the consolidation period a few bad ticks arrived, would the consolidated 10 minute bar be constructed using the available data and would that bar be pumped in as normal?

Thanks

Update Backtest

4 Comments

Mike G

, ,

Accepted Answer

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Alexandre Catarino

Hi Mihail,

Say you add XYZ with second resolution into your user universe.
If from 9:30:00 to 9:30:05 there is no data from XYZ, OnData event handler will not be triggered and no trades with XYZ will be executed in this period.

We are always looking into providing robust data sanity checks to spot bad data in our database.

Jared Broad

There shouldn't ever be "bad" data; but if there is no trading there is definitely a gap until the first data point. After this the data is copied forward when there's nothing new (fillforward mode). Consolidators data is pushed out when the consolidated time passes the bar end time.

Dean Pignon

293 ,

Here's an example of what I would call bad data:

SPY options, minute resolution, at 2010-05-06 14:48,

SPY 100522P00131000 (put at $131 strike with two weeks to expiry) price = $100,009.51
SPY price = $111.23

Both prices were obtained using `SecurityHolding.Price`. This is the only such glitch I have come across so far; there may be others that are not as obvious or in data that I have not used.

Jared Broad

Hi Dean, since 2016 we've implemented the data explorer which lets you browse the data; located any issues, and submit them to a list to be fixed. You can subscribe to the discussion and be notified when it's patched. Please see more here:

https://www.quantconnect.com/data/tree