Back

SPY Options (minute) missing files

Ticker SPY
Security Type Option
Market USA
Resolution Minute
Status In progress

Corrupted Files Missing Files Missing Points Wrong Values TickerSPYSecurity TypeOptionMarketUSAResolutionMinuteStatusWaiting Review

Hi,

First up, thanks for making QuantConnect - you guys and gals are awesome!

Ok, so I've been tracking down an issue where sometimes when I try to close an options leg, the order would come back as Invalid with the options contract symbol being marked as "non-tradable".  After some head scratching, I eventually figured out some data was missing.  For example, for SPY minute options on 2015-04-28, there are call options data, but there are no put options.

You can see in Data Expolorer that "/data/option/usa/minute/spy/20150427_quote_american.zip/*_put_*.csv" is missing.

So with that, I hacked up a little agorithm to try and detect when there was missing files like this.  I've only looked at SPY for monthly options with expiries less then 60 days to expiration.  It works by once a day every day at 2 hours after market open, check to see if theres an options chain, and if there is, check if there is at least 1 ATM call and 1 ATM put for each of the expiry dates it's seen before.  It's not foolproof, but it does highlight where theres missing data.

When run from 2012-01-01 to present, I get a few missing days:

On 2012-04-12 no options chain for ?SPY
On 2013-10-14 SPY options for expiry=2013-11-16 is missing: calls puts
On 2014-09-03 SPY options for expiry=2014-10-18 is missing: calls puts
On 2015-04-27 SPY options for expiry=2015-05-15 is missing: puts
On 2015-04-27 SPY options for expiry=2015-06-19 is missing: puts
On 2015-04-28 SPY options for expiry=2015-05-15 is missing: puts
On 2015-04-28 SPY options for expiry=2015-06-19 is missing: puts
On 2016-03-23 no options chain for ?SPY
On 2016-03-28 no options chain for ?SPY
On 2016-03-29 no options chain for ?SPY
On 2016-04-08 no options chain for ?SPY
On 2016-04-13 no options chain for ?SPY
On 2016-04-21 no options chain for ?SPY
On 2017-06-12 no options chain for ?SPY
On 2017-06-15 no options chain for ?SPY
On 2018-01-10 SPY options for expiry=2018-01-19 is missing: puts
On 2018-01-10 SPY options for expiry=2018-02-16 is missing: puts
On 2018-01-10 SPY options for expiry=2018-03-16 is missing: puts
On 2018-02-08 SPY options for expiry=2018-02-16 is missing: calls
On 2018-02-08 SPY options for expiry=2018-03-16 is missing: calls
On 2018-03-06 SPY options for expiry=2018-03-16 is missing: calls
On 2018-03-06 SPY options for expiry=2018-04-20 is missing: calls
On 2018-03-07 SPY options for expiry=2018-03-16 is missing: puts
On 2018-03-07 SPY options for expiry=2018-04-20 is missing: puts
On 2018-06-14 no options chain for ?SPY
On 2018-07-23 no options chain for ?SPY

 

When run prior to 2012 back to 2007 (which looks like the start of the data), there apears to be many more gaps:

On 2007-01-03 no options chain for ?SPY
On 2007-01-04 no options chain for ?SPY
On 2007-01-05 no options chain for ?SPY
On 2007-01-08 no options chain for ?SPY
On 2007-01-09 no options chain for ?SPY
On 2007-01-10 no options chain for ?SPY
On 2007-01-11 no options chain for ?SPY
On 2007-01-12 no options chain for ?SPY
...
... no options chain for any of these days ...
... (which is unfortunate given this would be a great time period for backtesting)
...
On 2008-12-22 no options chain for ?SPY
On 2008-12-23 no options chain for ?SPY
On 2008-12-24 no options chain for ?SPY
On 2008-12-26 no options chain for ?SPY
On 2008-12-29 no options chain for ?SPY
On 2008-12-30 no options chain for ?SPY
On 2008-12-31 no options chain for ?SPY
On 2009-01-02 no options chain for ?SPY
On 2009-01-05 no options chain for ?SPY
On 2009-01-06 SPY options for expiry=2009-02-20 is missing: calls
On 2009-01-07 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-08 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-09 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-12 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-13 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-14 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-14 SPY options for expiry=2009-01-16 is missing: calls puts
On 2009-01-15 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-15 SPY options for expiry=2009-01-16 is missing: calls
On 2009-01-16 SPY options for expiry=2009-02-20 is missing: calls
On 2009-01-16 SPY options for expiry=2009-03-20 is missing: calls
On 2009-01-20 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-20 SPY options for expiry=2009-03-20 is missing: calls
On 2009-01-21 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-21 SPY options for expiry=2009-03-20 is missing: calls
On 2009-01-22 SPY options for expiry=2009-02-20 is missing: calls
On 2009-01-22 SPY options for expiry=2009-03-20 is missing: calls
On 2009-01-23 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-23 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-01-26 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-26 SPY options for expiry=2009-03-20 is missing: puts
On 2009-01-27 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-28 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-29 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-29 SPY options for expiry=2009-03-20 is missing: puts
On 2009-01-30 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-01-30 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-02 SPY options for expiry=2009-02-20 is missing: calls puts
On 2009-02-02 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-03 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-04 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-05 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-05 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-06 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-06 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-09 SPY options for expiry=2009-02-20 is missing: calls
On 2009-02-09 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-10 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-10 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-11 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-11 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-12 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-12 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-12 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-13 SPY options for expiry=2009-02-20 is missing: calls
On 2009-02-13 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-17 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-17 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-17 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-18 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-18 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-18 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-19 SPY options for expiry=2009-02-20 is missing: puts
On 2009-02-19 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-19 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-20 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-20 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-23 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-23 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-24 SPY options for expiry=2009-03-20 is missing: puts
On 2009-02-24 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-25 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-26 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-26 SPY options for expiry=2009-04-17 is missing: puts
On 2009-02-27 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-02-27 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-02 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-02 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-03 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-03 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-04 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-04 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-05 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-05 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-06 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-06 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-09 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-09 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-10 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-10 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-11 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-11 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-12 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-12 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-13 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-13 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-16 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-16 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-17 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-17 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-18 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-18 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-19 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-19 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-20 SPY options for expiry=2009-03-20 is missing: calls puts
On 2009-03-20 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-23 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-24 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-25 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-26 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-27 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-30 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-03-31 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-01 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-02 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-03 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-06 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-07 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-08 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-09 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-13 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-14 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-15 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-16 SPY options for expiry=2009-04-17 is missing: calls puts
On 2009-04-17 SPY options for expiry=2009-04-17 is missing: calls puts
On 2011-05-02 no options chain for ?SPY
On 2011-05-03 no options chain for ?SPY
On 2011-05-04 no options chain for ?SPY
On 2011-05-05 no options chain for ?SPY
On 2011-05-06 no options chain for ?SPY
On 2011-05-09 no options chain for ?SPY
On 2011-05-10 no options chain for ?SPY
On 2011-05-11 no options chain for ?SPY
On 2011-05-12 no options chain for ?SPY
On 2011-05-13 no options chain for ?SPY
...
... no options chain for any of these days ...
...
On 2011-11-28 no options chain for ?SPY
On 2011-11-29 no options chain for ?SPY
On 2011-11-30 no options chain for ?SPY
On 2011-12-01 no options chain for ?SPY
On 2011-12-02 no options chain for ?SPY
On 2011-12-05 no options chain for ?SPY
On 2011-12-06 no options chain for ?SPY
On 2011-12-07 no options chain for ?SPY
On 2011-12-13 no options chain for ?SPY
On 2011-12-15 SPY options for expiry=2011-12-17 is missing: calls puts

Update Backtest







Attached is the algorithm I used to detect the missing data above.

As mentioned above, I've only looked at SPY options <60 days to expirtion.  In future when I look at other underlyings, if there's any missing data with them, I'll post separately.

0


Thanks Michael Lee - we'll look into it this week.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thank you Michael for such analysis and the algorithm!

The data since 2016 up to today is being patched right now, but as patching option data is a slow process, it'll take a couple of days to have all data in place.
Data previous 2016 is an even slower process, but once the current job finish, we'll fix those missing dates too.

However, there is a caveat, if you check our vendor's website, it says historical data "from January 2012". So even when there is data available since 2007, data previous 2012 probably have missing observations.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


JayJayD has marked this issue as in progress.

0

Thanks!

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed