Back

Anomalous Change in Backtesting vs Live Results

Hi All,

I've been running a live algo on the QC environment since Oct 23 2019.

It's current equity curve looks like this:

Live.png

However, when I now backtest the very same code. I get an equity curve like this:

Backtest.png

The code is identical, bar a SetStartDate parameter to align with the live start date. I did not have this issue with any previous backtesting, as recently as 3 weeks ago. 

Has QC recently changed it's C# backtesting engine? Any help would be greatly appreciated.

Update Backtest







0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Can't see your uploaded images DeltaQuant but looks like I'm not the only one. Is your strategy high resolution/intraday? This change appears to have happened in the last 24 hours; my main strat that showed profit over the past week now swings to a loss over the same period in backtesting. I confirmed this by copying old code from that very same backtest I ran only yesterday that showed profit - the exact same code re-run today also shows a loss! Likewise the out-of-sample backtest also shows losses during recent days where my strategy was profitable. 

I dug into the orders log to compare the old and new side by side and it seems my algo is not exiting and taking profits on positions it would have previously. Very strange! I plan to go to minimum trade size in live and letting it run to see confirm if this is an issue with the backtesting engine (unless QC staff gets back to us here first)...

 

 

0

Hey there! We quietly rolled out L1 data last night. You can read more about the impacts to your algorithm performance and why those changes have happened in the L1 Discussion thread

The short answer is due to the impact of spreads market orders will now fill slightly worse than they did before. This makes it more realistic to what you'd receive at a brokerage. You can try using limit orders to improve the performance but the execution is not guaranteed. 

We are also in the process of adding the same quote data to the live feed to improve the realism of paper fills. This is excepted to be launched in about 2 weeks. 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thank you both for the replies. 

Patrick, I'm glad to hear that I'm not the only one experiencing this issue. The algo averages about 2-4 trades per day and is not intraday specific. It's based on certain entry & exit market conditions. It is frustrating when the goal posts move, as it undermines confidence in the overall accuracy of the platform.

Jared, the algorithm already uses Limit orders, so there may be a bug in the update somewhere? I'll explore the L1 thread to see if that sheds some light on the recent change in execution.

0

Hi DeltaQuant, 

There are no changes to other parts of LEAN. Perhaps the spread has also reduced the likelihood of your limit orders filling. We're happy to assist your debugging if you'd like to submit a support ticket or post a backtest here.

I appreciate your concerns about undermining trust in the platform. We announced the L1 discussion thread about 2 months ago and posted regular updates on its progress. We attempted to deploy it unsuccessfully twice and rolled back to preserve accurate backtesting. To the best of our knowledge, the platform is more accurate than ever before. 

We spent about 6 months processing, reviewing and standardizing the data with a full-time engineer. It was a huge investment for the community. I highly recommend submitting a support ticket if you can so we can analyze it for you and get to the bottom of your differences.

Best
Jared

https://www.quantconnect.com/forum/discussion/7226/coming-soon-l1-equity-quote-data/p1

 

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed