Back

Coming Soon - L1 Equity Quote Data!

Hi All!

We're excited to share that coming soon there will be L1 quote data automatically included in backtests for US equity stocks. We are also updating the historical and live data feed to be based on the SIP; the aggregated tick feed from all the exchanges.

We strive to make our backtests as close as possible to live trading, but this modeling is difficult without the full volume and a realistic spread. With the new data sources, it should be a significant advance in the realism of the backtests. The new source would be identical to paper trading, allowing your live paper trading to be more realistic than ever.

If you are interested in the engineering please feel free to subscribe to follow along. Once the code is ready and data in place we'll merge the pull-request and hot-swap the data over. There should be no downtime. I will also comment on this discussion thread once the merge is complete.

Important Note #1: Because the backtests will now have spread; your backtest performance will likely be worse than it was before! I understand this will be upsetting but please see it as a more realistic reflection of the model. It means if you buy and sell in the same second, it will constantly lose the spread between the equity bid and ask. This cost will be significant if you are trading quickly or on low liquidity assets.

Important Note #2: Quote data is approximately 3x the amount of data. Although the processing speed of LEAN will get faster in "data points per second" -- the absolute wall-clock time for the backtest will increase. We've got 2-3 massive engineering leaps up our sleeves which should offset that delay by running 300% faster... there will be 1-2 weeks where the backtest will be slower while we install those new engineering advances. 

We expect this upgrade to happen in the next week. Thank you for supporting QuantConnect! Here's to an awesome 2020! 

Best Regards,

Jared

Update Backtest





The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Very exciting and wonderful update guys!
The more data the merrier. Much preferred to be upset now then later!!

3

I totally agree with Ted. You guys are doing a great job!

1

Thanks Arthur! =)

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Very interested to use this new data.

How will this effect the alpha markets existing algos? It's likley that algos that were previous thought to be suitable may no longer be?

I guess having an overall, higher quality/realistic algo offering would amount to a better QC reputation amongst funds.

1

Looking forward to the performance impact on my algo's, exciting! Short side question relating to this: what will be the impact of including this data on the Alpha slippage model, is it still relevant or will this be depricated?

best,

Dirk

0

The alphas in the market will be stopped and redeployed (with their history intact but a mark on curve noting the change in the data source). If they continue to perform well with the new slippage factored in they'll be left to run. 

Separately we're building in an alpha-system-wide, automatic "stop" mechanism for alphas who no longer perform according to backtest. This should stop any which fail due to the new slippage.

The AlphaStream slippage model will be reduced but not eliminated. Even after accounting for the bid-ask spread there is still slippage on the filling of a large order. This slippage should be made proportional to the size of the trade. 

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Fantastic news. Overly optimistic fills was really one of the main time killers when starting out (before knowing better). Not to mention all other potential uses. :-)

Will it be possible to find e.g. average spread in universe selection to e.g. filter out securities with too wide spreads?

2

Cool idea Petter; once installed we can start exploring expanding coarse universe to factor in spreads.

2

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey Jared Broad 
That is a very good enhancement! It is so important (at time for Alpha's authors and for QuantConnect!) to prove that the Alpha on the marketplace are not profitting from any data anomally to make the QC Alpha Streams marketplace the reference!

2

Looking forward to it

0

Hi Jared! Any news on the progress, or an ETA, I'm so freaking excited to see what it does to my backtests : )

 

Best,

Dirk

0

=D Thanks Dirk getting there. We hit a roadblock in the final deployment of the data and needed to rebuild the entire 40TB data set. It's now processed and we're re-running the production update process to ensure it is updated each day, on time for the universe selection in live trading.

We're optimistic it'll be ready this week. 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey Jared! 

Hope you are well, and that this improvement will soon be effective!

Do you know when we will be able to try and test this L1 Equity Quote Data?

Best regards, and keep up the good work!

0

Hey Laurent! Getting there! We've green-lit the pull request and are not making the tests to ensure the redundant data processing is online from day-1 as well. We currently have full redundancy in the data stack and would like to keep it that way.

-----

One interesting change was the new coarse universe data dollar volume ranks change due to the new rules of what ticks will be included in the final volume math. Previously FINRA ticks were excluded from bars as they are not tradable (FINRA feed is post-trade reporting). This generated thousands of support tickets and forum questions from people wondering why we didn't match Yahoo.

We've opted to include this volume now to quiet these questions; in the hope that the more complete representation of volumes will improve those strategies sensitive to dollar-volume rankings; and technical indicators that require trade volume. 

Since the new data starts in 2007 we'll also go back and reprocess all the 1998-2007 data with these new tick rules to match the SIP feed data. This will ensure consistent ranking behavior from 1998-present.

Important Note #3: Dollar volume rankings of stocks will change with this new data. The analysis of the rank changes shows smaller ETFs (e.g. Brazilian Stock Market ETF) which previously had high ranks are now displaced by stock counterparts. We can see the deviation of the two data sources from last week's coarse universe within the plots below:

677_1581365183.jpg

For a single day the differences are more obvious:

677_1581365244.jpg

Which if you look at in detail is mostly movements of obscure assets out of the top 100, and popular assets into the top 100. We think this is a good change and although a breaking change, makes the data better overall. 

The further you go back in time the differences become less apparent. Presumably due to lower dark-pool volume which is reported to FINRA; this is the same scatter from 2008:

677_1581365312.jpg

-----------

Ultimately the most important thing is that our backtesting matches live-trading. So once the new live SIP feed is installed we'll also have the FINRA dark pool data there for algorithm consumption! We'll email all the impacted live users and recommend they re-backtest their strategies to confirm the coarse selection process still run as they expect or to take their strategies offline for review.

 

2

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hi Jared: I'm super excited about this addition. Is this data in full effect for backtesting and live trading now? If not, how far out is it expected to be?

Again, super exciting stuff! Thank you and best regards

0

Thanks David we're excited! The backtesting can be this week. The live feed paperwork is signed and the prototype live feed is in place. We're doing load tests on the live data feed now to ensure the new increases in tick volume won't overwhelm the running algorithms. I'd ideally like to hold off deploying the backtesting data until we can swap the live feed at the same time. This might make it after market close Friday.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed