Summary
Augment a backtest by assessing its variability.
Description
Testing quantitative trading strategies is necessary, but difficult. There is the danger of data snooping — that is, thinking that the best of a large number of tested strategies is better than it really is. Another danger is to find the same thing that lots of others have found. August 2007 showed how hazardous that can be.
Random portfolios can help relieve both of these problems. They give you more stringent criteria with which to accept a strategy. They also allow you to see more ephemeral signals, and hence help you avoid doing what everyone else is doing.
A backtest starts with some initial portfolio at the start of the time period. Trades use the reputedly market-beating signal while obeying a set of constraints. The result is a return over the period. We can learn the significance of that return by imitating the backtest but leaving the signal out. We start with the same initial portfolio, and do random trades throughout the period that obey the constraints, but pay no attention to the signal.
Figure 1: QQplot of performance percentiles
It is not good practice to use just one starting portfolio — you could be the victim of either good or bad luck. Figure 1 shows a strategy performance relative to random paths for twenty different starting portfolios. A performance percentile of 0 means that no random paths did better than the backtest, 100 means all random paths did better.
If the strategy were doing nothing at all, we’d expect the points to be roughly along the diagonal line. In this case there is a slight indication that the strategy is doing something positive. That there are a lot of points at the extremes indicates the strong power of the method.
Figure 2: Mean strategy outperformance
Once you have a sense that the strategy truly outperforms, you want to know by how much — is it trivially outperforming or seriously outperforming? A plot like Figure 2 can help you with that. This looks at the strategy return minus the mean of the returns of the random paths. Figure 2 is plotting the mean of that across the twenty runs. Figure 1 is for the first quarter 0f 2008, so at day 61.
Real-time
This technique need not be limited to backtests. You can use it in real-time as well to see how your strategy is currently performing.
