“If you can’t come up with a pretty common sense explanation that you can describe to your 12-year-old niece or nephew. Then chances are, you’re simply data mining. Coming up with something that won’t exist in the future.”said Mebane Faber (@mebfaber), Portfolio Manager at Cambria Investments.

When Mebane first started in quant trading he thought he had found the “holy grail” model by analyzing its historical returns. Mebane says extrapolating historical returns to the present is a common mistake made by new quants which can be fixed if you know why your model works. “Most traders that are older have a lot of battle scars either from real money or paper trading portfolios that performed differently than the historical model showed.” says the experienced Mr. Faber.

So what is Mebane’s advice to new quants looking to perfect their backtesting strategy: “come up with a system, or multiple systems (which I think is more important) that fits your personality but is also robust over time“. Wise words from a King of quantitative trading.

2. Use Blind Data to Improve Your Strategy

Optimizing a strategy is for some a process that is ongoing and for some a process that produces such low results that it is abandoned. Deepak Shenoy(@deepakshenoy), Co-Founder of Capital Mind a big data analysis company, says it is a combination of both groups “I don’t like curve fitting so I tend to avoid over-optimization“. Instead he adds unrelated factors “like volume, open interest or options price sensitivity (vega) – or simply by using the knowledge of near term events to augment a system“.

Additionally, an important step for Deepak after he’s optimized, is to test on a blind data set for validation. “I test an optimized strategy on either different sets of stock data or different timeframe data… or both. To see how the system does in other timeframes.” This is before he moves the system to a lengthy paper trading schedule to further validate the model. Deepak’s perspective seems to be a focus on perfection, validation from multiple sources and comprehensive market regime testing. A worthy lesson for quants, be a perfectionist!

3. Decide Your Most Important Metrics Before

When you get to the performance evaluation segment of your strategy backtesting process, historical data is plentiful. “While it is pretty standard to look at popular performance statistics like the Calmar Ratio, Sharpe Ratio, CAGR, or MaxDD, they only reveal a small picture of how the model performed” says Michael Guan, an Associate with Macquarie Group. Michael says he focuses on additional measures like Positive Rolling 12 month periods, average drawdown, Annual Returns, average profit per trade which all provide a “more multi-dimensional view through time“.

Michael is a true believer in a multi-dimensional backtesting process. He explained his viewpoint eloquently: “backtesting is an integral part of trading model development. Making as few assumptions as possible and testing on a wide variety of assets ensures statistical robustness“. Many quants run a backtest with no preconceived notion of the data they want and this creates an issue in determining what data is valuable. Follow Michael’s tip and make a list of the variables that matter to you. And data has more value to you, is the Sharpe Ratio worth more to you than the P:L ratio? You should know this before you start backtesting.

4. Make Sure You Are Looking At The Right Data

6. Don’t Fix Everything Now, Be Ready for A Regime Change

Of course, in any extensive and well conducted backtesting process you want to test out your strategy on as many companies as possible in as many timeframes as possible. Lars Foleide (@zyron), who teaches Quantitative Finance at UC Berkeley says “What I focus on is the abnormal events, not the average. Because abnormal events can’t be backtested for“. He says most quants are interested in how to churn money slowly and be always invested. However what Lars looks for is how to create a system that isn’t perfect for any market but is ready to be adjusted with the market.

Lars believes traditional quant trading is too robust and stubborn when it comes to adjusting algorithms. With volatile markets, not only should one’s strategy be continuously optimized based on historical results, according to Lars, a model should be optimized by monitoring current market events as well. “I’m looking for Black Swan events. How to predict them and how to have my system be ready for a regime change” says Mr. Foleide. Being open to the idea that your strategy might not work in 6 months’ time is a lesson Lars wants new quants to know and be prepared to adjust to.

7. When Do You Stop Optimizing Your Strategy

In conclusion, not all the tips above will be applicable to all strategies out there but the above insight should help in understanding the perspective of top quants as they approach strategy backtesting. The one key to strategy backtesting that all the experts we spoke to agreed with, was a focus on experimentation and continuous monitoring. We look forward to hearing your comments on the advice from our expert quants and hope the tips help you design better strategies!