In the latest episode of Better System Trader, the guest, Dave Walton, made a compelling case arguing AGAINST some of the most popular backtesting strategies: Specifically, out-of-sample testing, walk-forward analysis, and Monte Carlo analysis. He states these approaches are a very poor indicator of future results. He advocates instead for something he calls System Parameter Permutation. Basically, it's a way to model many possible permutations of the same system but with a range of different parameter values for its rules (ie. entry and exit rules). Then, by looking at all of the possible outcomes, you can draw better conclusions on the probability of how your strategy will work in the future.
You can listen to the podcast and read a much more detailed explaination here:
http://bettersystemtrader.com/051-dave-walton/
So, as a community, what do all of you think? Does he make some valid points? Most importantly, is this something that can be done with QuantConnect? What about the Lean Engine? If not, are there other platforms that offer this type of analysis?
Jared Broad
I agree with that philosophy; we wrote a blog which used it here: https://www.quantconnect.com/blog/rotating-inversely-correlated-assets-nifty-usdinr/
Basically any "magic number" you use; make sure you do lots of variations on the number, and see how it impacts your results. At least this way you're reasonably confident that its not sensitive / curve fitting.
We have a parameter collection; maybe we could automatically perform 100x backtest and see how sensitive each parameter is on a similar sharpe ratio chart. We're still trying to figure out what the Optimization GUI will look like.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Matthew Williams
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!