Ticker SPY
Security Type Option
Market USA
Resolution Minute
Status Resolved
Corrupted Files Missing Files Missing Points Wrong Values TickerSPYSecurity TypeOptionMarketUSAResolutionMinuteStatusWaiting Review

Hi,

First up, thanks for making QuantConnect - you guys and gals are awesome!

Ok, so I've been tracking down an issue where sometimes when I try to close an options leg, the order would come back as Invalid with the options contract symbol being marked as "non-tradable".  After some head scratching, I eventually figured out some data was missing.  For example, for SPY minute options on 2015-04-28, there are call options data, but there are no put options.

You can see in Data Expolorer that "/data/option/usa/minute/spy/20150427_quote_american.zip/*_put_*.csv" is missing.

So with that, I hacked up a little agorithm to try and detect when there was missing files like this.  I've only looked at SPY for monthly options with expiries less then 60 days to expiration.  It works by once a day every day at 2 hours after market open, check to see if theres an options chain, and if there is, check if there is at least 1 ATM call and 1 ATM put for each of the expiry dates it's seen before.  It's not foolproof, but it does highlight where theres missing data.

When run from 2012-01-01 to present, I get a few missing days:

On 2012-04-12 no options chain for ?SPY
On 2013-10-14 SPY options for expiry=2013-11-16 is missing: calls puts
On 2014-09-03 SPY options for expiry=2014-10-18 is missing: calls puts
On 2015-04-27 SPY options for expiry=2015-05-15 is missing: puts
On 2015-04-27 SPY options for expiry=2015-06-19 is missing: puts
On 2015-04-28 SPY options for expiry=2015-05-15 is missing: puts
On 2015-04-28 SPY options for expiry=2015-06-19 is missing: puts
On 2016-03-23 no options chain for ?SPY
On 2016-03-28 no options chain for ?SPY
On 2016-03-29 no options chain for ?SPY
On 2016-04-08 no options chain for ?SPY
On 2016-04-13 no options chain for ?SPY
On 2016-04-21 no options chain for ?SPY
On 2017-06-12 no options chain for ?SPY
On 2017-06-15 no options chain for ?SPY
On 2018-01-10 SPY options for expiry=2018-01-19 is missing: puts
On 2018-01-10 SPY options for expiry=2018-02-16 is missing: puts
On 2018-01-10 SPY options for expiry=2018-03-16 is missing: puts
On 2018-02-08 SPY options for expiry=2018-02-16 is missing: calls
On 2018-02-08 SPY options for expiry=2018-03-16 is missing: calls
On 2018-03-06 SPY options for expiry=2018-03-16 is missing: calls
On 2018-03-06 SPY options for expiry=2018-04-20 is missing: calls
On 2018-03-07 SPY options for expiry=2018-03-16 is missing: puts
On 2018-03-07 SPY options for expiry=2018-04-20 is missing: puts
On 2018-06-14 no options chain for ?SPY
On 2018-07-23 no options chain for ?SPY

 

When run prior to 2012 back to 2007 (which looks like the start of the data), there apears to be many more gaps:

On 2007-01-03 no options chain for ?SPY
On 2007-01-04 no options chain for ?SPY
On 2007-01-05 no options chain for ?SPY
On 2007-01-08 no options chain for ?SPY
On 2007-01-09 no options chain for ?SPY
On 2007-01-10 no options chain for ?SPY
On 2007-01-11 no options chain for ?SPY
On 2007-01-12 no options chain for ?SPY
...
... no options chain for any of these days ...
... (which is unfortunate given this would be a great time period for backtesting)
...
On 2008-12-22 no options chain for ?SPY
On 2008-12-23 no options chain for ?SPY
On 2008-12-24 no options chain for ?SPY
On 2008-12-26 no options chain for ?SPY
On 2008-12-29 no options chain for ?SPY
On 2008-12-30 no options chain for ?SPY
On 2008-12-31 no options chain for ?SPY
On 2009-01-02 no options chain for ?SPY
On 2009-01-05 no options chain for ?SPY
On 2009-01-06 SPY options for expiry=2009-02-20 is missing: calls
On 2009-01-07 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-08 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-09 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-12 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-13 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-14 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-14 SPY options for expiry=2009-01-16 is missing: calls puts
On 2009-01-15 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-15 SPY options for expiry=2009-01-16 is missing: calls
On 2009-01-16 SPY options for expiry=2009-02-20 is missing: calls
On 2009-01-16 SPY options for expiry=2009-03-20 is missing: calls
On 2009-01-20 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-20 SPY options for expiry=2009-03-20 is missing: calls
On 2009-01-21 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-21 SPY options for expiry=2009-03-20 is missing: calls
On 2009-01-22 SPY options for expiry=2009-02-20 is missing: calls
On 2009-01-22 SPY options for expiry=2009-03-20 is missing: calls
On 2009-01-23 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-23 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-01-26 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-26 SPY options for expiry=2009-03-20 is missing: puts
On 2009-01-27 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-28 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-29 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-29 SPY options for expiry=2009-03-20 is missing: puts
On 2009-01-30 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-30 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-02 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-02-02 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-03 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-04 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-05 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-05 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-06 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-06 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-09 SPY options for expiry=2009-02-20 is missing: calls
On 2009-02-09 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-10 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-10 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-11 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-11 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-12 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-12 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-12 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-13 SPY options for expiry=2009-02-20 is missing: calls
On 2009-02-13 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-17 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-17 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-17 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-18 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-18 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-18 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-19 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-19 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-19 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-20 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-20 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-23 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-23 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-24 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-24 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-25 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-26 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-26 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-27 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-27 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-02 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-02 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-03 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-03 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-04 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-04 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-05 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-05 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-06 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-06 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-09 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-09 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-10 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-10 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-11 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-11 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-12 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-12 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-13 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-13 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-16 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-16 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-17 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-17 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-18 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-18 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-19 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-19 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-20 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-20 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-23 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-24 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-25 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-26 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-27 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-30 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-31 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-01 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-02 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-03 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-06 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-07 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-08 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-09 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-13 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-14 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-15 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-16 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-17 SPY options for expiry=2009-04-17 is missing: calls puts
On 2011-05-02 no options chain for ?SPY
On 2011-05-03 no options chain for ?SPY
On 2011-05-04 no options chain for ?SPY
On 2011-05-05 no options chain for ?SPY
On 2011-05-06 no options chain for ?SPY
On 2011-05-09 no options chain for ?SPY
On 2011-05-10 no options chain for ?SPY
On 2011-05-11 no options chain for ?SPY
On 2011-05-12 no options chain for ?SPY
On 2011-05-13 no options chain for ?SPY
...
... no options chain for any of these days ...
...
On 2011-11-28 no options chain for ?SPY
On 2011-11-29 no options chain for ?SPY
On 2011-11-30 no options chain for ?SPY
On 2011-12-01 no options chain for ?SPY
On 2011-12-02 no options chain for ?SPY
On 2011-12-05 no options chain for ?SPY
On 2011-12-06 no options chain for ?SPY
On 2011-12-07 no options chain for ?SPY
On 2011-12-13 no options chain for ?SPY
On 2011-12-15 SPY options for expiry=2011-12-17 is missing: calls puts