Strategy backtesting is a mix of art and science. Quants who rely too much on science will fall victim to the infamous Curve Fitting phenomenon. While some quants who overcompensate on their artistic balance will create disillusioned theories that back their models. We created this post to compile leading quants’ perspectives on strategy backtesting covering tools, tips and how to avoid common mistakes made while strategy backtesting.
1. Have A Common Sense Idea of Your Model
“If you can’t come up with a pretty common sense explanation that you can describe to your 12-year-old niece or nephew. Then chances are, you’re simply data mining. Coming up with something that won’t exist in the future.”said Mebane Faber (@mebfaber), Portfolio Manager at Cambria Investments.
When Mebane first started in quant trading he thought he had found the “holy grail” model by analyzing its historical returns. Mebane says extrapolating historical returns to the present is a common mistake made by new quants which can be fixed if you know why your model works. “Most traders that are older have a lot of battle scars either from real money or paper trading portfolios that performed differently than the historical model showed.” says the experienced Mr. Faber.
So what is Mebane’s advice to new quants looking to perfect their backtesting strategy: “come up with a system, or multiple systems (which I think is more important) that fits your personality but is also robust over time“. Wise words from a King of quantitative trading.
2. Use Blind Data to Improve Your Strategy
Optimizing a strategy is for some a process that is ongoing and for some a process that produces such low results that it is abandoned. Deepak Shenoy(@deepakshenoy), Co-Founder of Capital Mind a big data analysis company, says it is a combination of both groups “I don’t like curve fitting so I tend to avoid over-optimization“. Instead he adds unrelated factors “like volume, open interest or options price sensitivity (vega) – or simply by using the knowledge of near term events to augment a system“.
Additionally, an important step for Deepak after he’s optimized, is to test on a blind data set for validation. “I test an optimized strategy on either different sets of stock data or different timeframe data… or both. To see how the system does in other timeframes.” This is before he moves the system to a lengthy paper trading schedule to further validate the model. Deepak’s perspective seems to be a focus on perfection, validation from multiple sources and comprehensive market regime testing. A worthy lesson for quants, be a perfectionist!
3. Decide Your Most Important Metrics Before
When you get to the performance evaluation segment of your strategy backtesting process, historical data is plentiful. “While it is pretty standard to look at popular performance statistics like the Calmar Ratio, Sharpe Ratio, CAGR, or MaxDD, they only reveal a small picture of how the model performed” says Michael Guan, an Associate with Macquarie Group. Michael says he focuses on additional measures like Positive Rolling 12 month periods, average drawdown, Annual Returns, average profit per trade which all provide a “more multi-dimensional view through time“.
Michael is a true believer in a multi-dimensional backtesting process. He explained his viewpoint eloquently: “backtesting is an integral part of trading model development. Making as few assumptions as possible and testing on a wide variety of assets ensures statistical robustness“. Many quants run a backtest with no preconceived notion of the data they want and this creates an issue in determining what data is valuable. Follow Michael’s tip and make a list of the variables that matter to you. And data has more value to you, is the Sharpe Ratio worth more to you than the P:L ratio? You should know this before you start backtesting.
4. Make Sure You Are Looking At The Right Data