By: Simon Burns
Quant Development Intern09.08.2013
7 Tips For Fixing Your Strategy Backtesting a Q&A With Top Quants
Strategy backtesting is a mix of art and science. Quants who rely too much on science will fall victim to the infamous Curve Fitting phenomenon. While some quants who overcompensate on their artistic balance will create disillusioned theories that back their models. We created this post to compile leading quants’ perspectives on strategy backtesting covering tools, tips and how to avoid common mistakes made while strategy backtesting.
1. Have A Common Sense Idea of Your Model
“If you can’t come up with a pretty common sense explanation that you can describe to your 12-year-old niece or nephew. Then chances are, you’re simply data mining. Coming up with something that won’t exist in the future.”said Mebane Faber (@mebfaber), Portfolio Manager at Cambria Investments.
When Mebane first started in quant trading he thought he had found the “holy grail” model by analyzing its historical returns. Mebane says extrapolating historical returns to the present is a common mistake made by new quants which can be fixed if you know why your model works. “Most traders that are older have a lot of battle scars either from real money or paper trading portfolios that performed differently than the historical model showed.” says the experienced Mr. Faber.
So what is Mebane’s advice to new quants looking to perfect their backtesting strategy: “come up with a system, or multiple systems (which I think is more important) that fits your personality but is also robust over time“. Wise words from a King of quantitative trading.
2. Use Blind Data to Improve Your Strategy
Optimizing a strategy is for some a process that is ongoing and for some a process that produces such low results that it is abandoned. Deepak Shenoy(@deepakshenoy), Co-Founder of Capital Mind a big data analysis company, says it is a combination of both groups “I don’t like curve fitting so I tend to avoid over-optimization“. Instead he adds unrelated factors “like volume, open interest or options price sensitivity (vega) – or simply by using the knowledge of near term events to augment a system“.
Additionally, an important step for Deepak after he’s optimized, is to test on a blind data set for validation. “I test an optimized strategy on either different sets of stock data or different timeframe data… or both. To see how the system does in other timeframes.” This is before he moves the system to a lengthy paper trading schedule to further validate the model. Deepak’s perspective seems to be a focus on perfection, validation from multiple sources and comprehensive market regime testing. A worthy lesson for quants, be a perfectionist!
3. Decide Your Most Important Metrics Before
When you get to the performance evaluation segment of your strategy backtesting process, historical data is plentiful. “While it is pretty standard to look at popular performance statistics like the Calmar Ratio, Sharpe Ratio, CAGR, or MaxDD, they only reveal a small picture of how the model performed” says Michael Guan, an Associate with Macquarie Group. Michael says he focuses on additional measures like Positive Rolling 12 month periods, average drawdown, Annual Returns, average profit per trade which all provide a “more multi-dimensional view through time“.
Michael is a true believer in a multi-dimensional backtesting process. He explained his viewpoint eloquently: “backtesting is an integral part of trading model development. Making as few assumptions as possible and testing on a wide variety of assets ensures statistical robustness“. Many quants run a backtest with no preconceived notion of the data they want and this creates an issue in determining what data is valuable. Follow Michael’s tip and make a list of the variables that matter to you. And data has more value to you, is the Sharpe Ratio worth more to you than the P:L ratio? You should know this before you start backtesting.
4. Make Sure You Are Looking At The Right Data
Blake Woodard of RLF Capital Management doesn’t like the way most quants backtest. “When building and backtesting in C++, your results come out as static data“. He equates this type of backtesting as going through your backtesting blind and prefers when “you can see every cell and action as you break it down“. He does this because he says it’s really good for figuring out the logic behind the model.
Some UX tools are helping to give quants this ability but Blake still uses advanced Excel plugins to see how each section of his algorithm is affecting results. Transparency is key for Blake.
5. Don’t be afraid to Experiment with Tools
Spencer Connaughton (@spenc3rc) is a self-taught 21 year old quant and Managing Director for Archivolt Partners. His learning curve brought him from using C++, then Bloomberg and Mathematica. His advice? Each tool or language has it’s own pros and cons so experiment with different ways to build the same algorithm.
For example, he ended up building a Wolfram Alpha Mathematica model. He explains it as “my Mathematica model was also really, really valuable because Mathematica lets you play with different variables one at a time and isolate elements, draw graphs and visualize your system“
By moving between different languages and tools, Spencer got exactly the value from each system that he wanted. He puts it this way, “I prototyped the system in Mathematica and built my own backtesting system first in C++. I then moved everything over to Bloomberg formulas [for its robustness]” A variety of tools for a variety of results and insights. Be thorough and try them all says Spencer. Don’t be lazy could be another way of phrasing it to new quants, its worth learning each new tool.
6. Don’t Fix Everything Now, Be Ready for A Regime Change
Of course, in any extensive and well conducted backtesting process you want to test out your strategy on as many companies as possible in as many timeframes as possible. Lars Foleide (@zyron), who teaches Quantitative Finance at UC Berkeley says “What I focus on is the abnormal events, not the average. Because abnormal events can’t be backtested for“. He says most quants are interested in how to churn money slowly and be always invested. However what Lars looks for is how to create a system that isn’t perfect for any market but is ready to be adjusted with the market.
Lars believes traditional quant trading is too robust and stubborn when it comes to adjusting algorithms. With volatile markets, not only should one’s strategy be continuously optimized based on historical results, according to Lars, a model should be optimized by monitoring current market events as well. “I’m looking for Black Swan events. How to predict them and how to have my system be ready for a regime change” says Mr. Foleide. Being open to the idea that your strategy might not work in 6 months’ time is a lesson Lars wants new quants to know and be prepared to adjust to.
7. When Do You Stop Optimizing Your Strategy
“Never” says Michael Guan from Macquarie Group, “strategy development is a continual process, when a new concept has been tested and proven to add value, then it should be incorporated into the strategy.” He adds there are ways to improve backtest performance that don’t necessarily add to CAGR but rather reduce other parameters like drawdown, etc. “The general rule of thumb is to search for improvements on a strategy that make it less dependent on your parameter selection“. So by understanding how your Sharpe Ratio effects your drawdown and other variables, you can optimize your strategy much better. This is a skill that is acquired over time says Michael, but be aware that changing your code to increase/decrease a variable can easily turn your refinement process into a game of algorithmic strategy “Whac-A-Mole“!
In conclusion, not all the tips above will be applicable to all strategies out there but the above insight should help in understanding the perspective of top quants as they approach strategy backtesting. The one key to strategy backtesting that all the experts we spoke to agreed with, was a focus on experimentation and continuous monitoring. We look forward to hearing your comments on the advice from our expert quants and hope the tips help you design better strategies!