Back

Different Backtest Results for same Algorithm?

Hi everyone, 

I noticed something strange. I get different backtest results for identical algorithms. Start Date and End Date are fixed, so that's not the reason here. 

At first, I suspected that there was a bug somewhere in my code, which leads to a non-deterministic behaviour of the algorithm. But I couldn't find anything. Then I found out that sometimes NaN values appeared for particular securities in my dataframes. So the algorithm couldn't find any data for this security. This in itself is not a big deal and could be intercepted by a "data.ContainsKey(symbol)" for example. What bothers me more is that it finds data one time and another time it doesn't. As if it depends on coincidence... It looks to me a little bit like a non-deterministic behaviour is created somewhere under the hood. And I don't like that at all because it would undermine confidence in QC as a platform for backtesting and live trading. 

We obviously expect that the same algorithm will always produce the same results. If this is not the case, then backtest results lose all meaning. 

I don't want to share the original code here because it's intended for the alpha streams marketplace. But I created a minimalistic example with the security "INTU" which reproduces the error. 

Please perform multiple backtests to get different results. For me this was usually the case after 3-5 runs. 

I really hope that I just missed something and that there is a simple explanation for it. 

I am grateful for any support. 

Best regards, 

Arthur

Update Backtest








0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


I'm also experiencing this issue, Thank God I'm not the only one. Thought I was going insane. I hope this issue gets resolved soon. The issue started around a day ago or so for me, also with identical algorithms but different backtest results. Even with algorithms I hadn't touched going back as far as 4 months ago now suddenly have different backtest results. Problem definitely isn't in the code of the algorithm itself.

2

Thanks for attaching a backtest so we can debug it Arthur! Give us a few minutes and we'll have it fixed. I think it's likely one node in the cluster isnt behaving

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Experiencing the same issue here. I copy pasted the exact code I ran yesterday, 30th July 2020, and got different results. Glad to know its a common problem. My tests were ran on Community B-MICRO Server. Hope it helps to pinpoint the issue.

0

I'm wondering if this is a wide impact. I'm having issues with some of my backtesting as well. The start date set to 2020, 6, xx is triggering options for 2006; 2020, 7, xx triggers options for 2007, and so on. 

Thank you

0

Thanks all. We have identified a corrupt map file and are deploying the previous version.

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Malcolm Tok can you please share your tickers affected? So far it looks like just the INTU with a bad mapping.  

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Original tickers used in the backtests yesterday were SPY, TLT, GLD. Having problems with those  3 plus IVV, QQQ, IAU, IEF while debugging today.

0

We can only see an issue with INTU and a handful of other niche tickers nothing generally and nothing with those "big-name" tickers. Please submit a backtest to support if you believe it continues. I've run 20 backtests on those symbols and it was been deterministic 20 of 20.

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


We posted a data issue here for the impacted tickers. Thank you all for the reports.

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Ran the same code and got back my initial results. Still unsure why there was a difference yesterday but I'm glad it is gone. Thanks for the fix.

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed