Over-Optimisation or Curve Fitting
Over-Optimisation is probably one of the biggest problems across the board. Curve fitting or data mining is basically taking optimal inputs from the past and expecting them to continue to work in the future. For example, let’s use the turtle’s breakout. The turtles had a twenty day breakout. Trading that kind of strategy doesn’t work particularly well, so you might go back and optimise what’s the best breakout. You might find that rather than twenty days, it’s actually a hundred days. The problem with that is you’ve made that decision after you’ve realised the twenty days doesn’t work. You’re always chasing your tail when you optimise; people want the perfect strategy. They want the strategy that’s going to work in all market environments all the time; that doesn’t exist.
Second of all, it doesn’t have to exist. Take a look at traders with a thirty or forty year track record. They’re producing 15%, 16%, 17%, 18%, 19% and 20% return over a forty year period. Compounding over that time frame there is huge, you get very wealthy. Look at Warren Buffet, he’s a got a compounded return of 19% over the long term. For some unknown reason people think that professional traders make substantially more money –maybe some do, but it’s not necessary. Second of all they think professional traders must get it right, 80% or 90% of the time. Again, that’s a long way from the truth.
When people have these misconceptions, they tend to optimise and data mine strategies, thinking that, “this strategy has a 75% win rate so it’s a much more profitable strategy than this one that has 45% win rate.” That’s actually incorrect. It’s very easily proven that high win rate systems, unless they have a high frequency, don’t generate enough profit, or don’t generate more profit than something simple like a trend following strategy that has a higher dollar win per trade.
It’s natural for people to want to optimise a strategy to get that win rate very high. It sells books.
It’s more important to have a strategy that works reasonably well all of the time, rather than a strategy that works absolutely incredibly well, but only part of the time.
Warning signs that you may be over-optimising a strategy
The biggest warning sign is a strategy that works only in a single market. A lot of vendors will sell a strategy that trades soy beans, or trades BHP or trades the S&P 500. A single market system usually has a data mining or curve fit optimisation element. Be very weary of a single market system. If you’ve got a pattern that operates in a single market, if the pattern has any validity, it should work across a variety of different markets.
So the first thing I do when see a single market system that works particularly well is I’ll go and trade it on a bigger universe of stocks. Say, the S&P500. If it doesn’t work then the pattern is a curve fit and it’s going to fail –that’s the first thing. The second thing is the number of inputs into a system. Generally speaking, the more you filter a pattern or setup, the more you’re curve fitting to past data, and generally speaking the results will be too good to be true. That’s where you get the systems that are generating 40% annualised return with a 4% draw down: they don’t exist. That is because people continue to remove the bad parts of the system –the bad trades by filtering. Unfortunately, what happens in the future, bad trades will come from somewhere else, which you haven’t taken into account.
I wrote about a setup of trades with just one Gold ETF. That same setup failed dismally, as soon as you went and traded on S&P 500 stocks. Another warning sign would be, if you have a very specific universe of say, “this system only works”, or “this system has to be traded on these five ETF’s”. Now, there’s many ETF’s, so one would assume that the strategy is going to work on the vast majority of those ETF’s, rather than just five. Unless there’s some specific reason why those five were chosen, then there’s a very high chance that the strategy’s been curve fitted.
