Back

Do you trust your backtest? What is System Parameter Permutation and is it a better way?

In the latest episode of Better System Trader, the guest, Dave Walton, made a compelling case arguing AGAINST some of the most popular backtesting strategies:  Specifically, out-of-sample testing, walk-forward analysis, and Monte Carlo analysis.  He states these approaches are a very poor indicator of future results.  He advocates instead for something he calls System Parameter Permutation.  Basically, it's a way to model many possible permutations of the same system but with a range of different parameter values for its rules (ie. entry and exit rules).  Then, by looking at all of the possible outcomes, you can draw better conclusions on the probability of how your strategy will work in the future.  

You can listen to the podcast and read a much more detailed explaination here:
http://bettersystemtrader.com/051-dave-walton/

So, as a community, what do all of you think?  Does he make some valid points?  Most importantly, is this something that can be done with QuantConnect?  What about the Lean Engine?  If not, are there other platforms that offer this type of analysis? 

Update Backtest








I agree with that philosophy; we wrote a blog which used it here: https://www.quantconnect.com/blog/rotating-inversely-correlated-assets-nifty-usdinr/

Basically any "magic number" you use; make sure you do lots of variations on the number, and see how it impacts your results. At least this way you're reasonably confident that its not sensitive / curve fitting.

We have a parameter collection; maybe we could automatically perform 100x backtest and see how sensitive each parameter is on a similar sharpe ratio chart. We're still trying to figure out what the Optimization GUI will look like.

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed