In-sample and Out-of-Sample Testing

This tends to come back towards those that move towards optimisation. Optimisation is fine when it’s done correctly, but usually people do it incorrectly. The way I was taught was using in sample and out of sample data. You divide your data into three parts. Let’s go back to 2000. We’ve got fifteen years and we divide it into three. We take 2000-2005, 2005-2010 and 2010 till today, so you have three sets of data. Then you go to that middle set of data, 2005-2010. Theoretically you would build your system in and around that five year period of time only. You would do your optimisation tests and come out with a strategy that looked good. Then you take those exact rules from that five year period, and apply it to the first five year period and the last five year period. That is your out of sample data.

If those two out of sample periods are similar to that middle sample, then you’re onto something.
If you test your strategy in a really good period of time, for example 2004 to 2007, you were probably saying, “I’ve found the holy grail!” You have to look at it in a bigger picture perspective and understanding that the GFC-type events are going to happen, and a trend following strategy that’s long only is going to be impacted in a certain way.

I saw recently a quant blog in the U.S. criticise a mean reversion system that was long only because it had flat periods in it. So what? It’s a long only mean reversion strategy that switches itself off during bear markets. That’s not a problem with the strategy –it’s a defensive mechanism. Warren Buffett doesn’t have winning years every year – he doesn’t need to. You just need to be prepared for your equity curve to be a little bit flat sometimes.

I have always been taught to test with three periods of time. You take the middle time and do your testing on that, then apply those parameters to the first set and then the last set, and if they equal up then you’re onto something.