When is a system broken?

[JulianCohen]

I came across this post in another forum I am a member of. Some of you here might have seen it too. It made me think.

BTFD is “buy the fookin dip” in case you need it.

My thinking after reading this post, which is from a Market Wizard BTW, is that if he backtested the ‘new’ system that he has written due to the old one breaking down in 2012, then I would imagine anything before 2012 would have produced poor, or very poor results.

So here’s the thing….If you are backtesting a system and the results for 2000-2003 are poor with a big drawdown, but since then the system is really good. Would you trade it?

Two thoughts here…one is that if an idea has only just become valid in the past ten years or so, but wasn’t before then…if you backtest earlier then it will show poor results.

Second is that if you require your backtests to work well over the last twenty years, are you over optimising or are you just exposing the system to many different markets? (…and then maybe over optimising it)

I think this line of thought applies mainly to short term systems, MOCs or Swing systems, as longer term WTTs and Rotations are working off momentum and trend following so that is a different thing.

MOCs and Swing are working off many varied patterns of behaviour in the markets, and sometimes they change.

I’d appreciate your thoughts.

[GlenPeake]

Yeah… I saw the same post Julian.

So here’s the thing….If you are backtesting a system and the results for 2000-2003 are poor with a big drawdown, but since then the system is really good. Would you trade it?

Yes… I’d trade it.

I generally focus on the last 15 years or so for my backtests…I know that when I was building my Monthly Rotational systems (NDX100/ASX100) the Tech Wreck period saw some large ‘gut check’ drawdowns etc… I’d found I was probably trying to tune my RTN systems to ‘weed out’ i.e. optimize the Drawdown ‘out of the backtest’ for that 2000 period… which had a negative effect on more recent periods. i.e. me optimizing for that 2000 period, hurt the results overall…. therefore I just accepted it and focused on more recent years etc… I guess if you are of the opinion that Markets Evolve, then what we saw in the Tech Wreck was perhaps a black swan (and a NASDAQ in it’s infancy) etc…. And Black Swan’s come in different forms etc…. GFC, European Debt Crisis, Trump Election win in 2016 and COVID etc…. so there is plenty of (recent) periods where there is added VOLA to test against.

I’ll look at the late 1990’s and early 2000’s, just to see how it looks… but will tend to focus on the last 15 years i.e. 2005-2020 etc.

It probably comes down to personal preference….If you want you systems to look good for the last 30 years in a backtest and that gives you confidence in trading it etc then that’s what gives you confidence etc

It might also boil down to the system(s) working ‘Pretty Good, most of the time’, in the view of the person that is going to trade it.

I built my 2 Mean Reversion systems with the 2005-2020 backtest data… I just ran a backtest on 2000-2004 period….. it didn’t look pretty…. but for me I place more emphasis on the more recent data and why I recalibrate (my MRV systems) using the last 3-5 years of data etc. In addition we can stress test our systems using variables for OPEN/HIGH/LOW/CLOSE and other parameters etc…..which also provides another layer of verification/testing/confidence etc.

[JulianCohen]

I guess I’m thinking about it from an existential level as well. Do you think that if you take your system that works great from 2005-2015 and then tweak it so that it works for 2000-2005 as well, then that is over optimization?

[Nick Radge]

I also think you need to consider what that person was doing. Apparently his strategy died on the spot from what I understand – and that’s extremely odd. It literally went from profitable to a dead-set loser immediately, so was there optimisation/data mining going on?

Like Glen, the TLT Premium Portfolio had a large drawdown in 2000 – but it’s explainable. Either side of that event the system is excellent.

[GlenPeake]

Julian Cohen wrote:

Do you think that if you take your system that works great from 2005-2015 and then tweak it so that it works for 2000-2005 as well, then that is over optimization?

I guess it’s a personal choice…. I’d prefer to Optimize against more recent data, to give me the best chance of being in SYNC with the (more recent) market activity rather than Optimize against older data which ‘might’ be less relevant these days, but as long as the backtests look good for the last 10-15 years, I’m ok with that… that system will then suit my personality.

I confess I’ve struggled to build my MRV systems that backtest with nice consistent stats across a 25-30 year period etc… So I’ve just focused on the last 15 years.

[ScottMcNab]

Julian Cohen wrote:

I came across this post in another forum I am a member of. Some of you here might have seen it too. It made me think.

So here’s the thing….If you are backtesting a system and the results for 2000-2003 are poor with a big drawdown, but since then the system is really good. Would you trade it?

I’d appreciate your thoughts.

Do you need the system ? (ie sufficiently low correlation to existing systems or other systems reached capacity)

Do you have enough confidence in the robustness of the system going forward that when it hits hard times and hits its largest drawdown that you will keep trading it….or will there be an element of doubt ?

[JulianCohen]

Scott McNab wrote:

Do you need the system ? (ie sufficiently low correlation to existing systems or other systems reached capacity)

Do you have enough confidence in the robustness of the system going forward that when it hits hard times and hits its largest drawdown that you will keep trading it….or will there be an element of doubt ?

Do I need the system? Well I’m always testing systems to find new ones with low correlations to the old ones to add diversification, so yes.

Do I have confidence in the robustness….that’s the question. I am at a decision point here and trying to work my way through it.

One side…systems that work through 2000 to today, especially with short systems, as I have a few that seem to struggle 2000 to 2003/4 ish. Then they do great. So if I can get them to work 2000 to now then up until the last few days I was confident, but then I was thinking about over optimisation and it made me think about that a bit more. Plus some of them I just can’t get to improve over that period.

Other side, systems that have big and consistent drawdowns 2000 to 2004 then do very very well. If I traded those and the drawdown started would I turn them off?

Well I am now thinking that yes I probably would, but that is OK. It is not the only system I am running, it’s one of many. If I set a drawdown level where I say at that point it is possibly failing, or something is changing that I haven’t worked out yet, then switch it off until it corrects itself…or maybe never does.

I think that is probably the best way to approach it.

Typing it all out has answered the question for me I guess

[ScottMcNab]

When I asked “do you need it” I was thinking along the lines of “so many rabbit holes and so little time”. My concern when have little faith in a system is that will turn it off at the bottom of the drawdown and it could potentially cost you $$.

[JulianCohen]

The advantage of having a few systems is that one doesn’t become too attached to any particular system. So “need” doesn’t really apply. I’m not going to be in a position (I hope) where I need the system otherwise I don’t have one to trade, which can bring in all other sorts of emotional issues.

I just need it cause it’s maybe good…

[AlexanderLuong]

I usually test from 2006 cause Nick picked that year when he was talking to me on the phone once. This is my MR MOC Short I am running:

Looks ok, right?

Then I was bored one day and tested back to 1995:

Yikes! Scared me a little LOL…

Not sure what’s up with this 1995 data all 0:

But I’m trading it cause I feel like it’s probably a good idea to do both long + short and I don’t have anything else on the short side! I watched it make a ton of money and then lose it all in a day 3 times already! Lots of fun for a newbie trader!

So I’m just gonna ignore 1995-1996 because at the moment, I think the most important thing for me is just trading and following the process. Which I’ve been doing!

Edit: Oh woops, that’s set to 25 years to 1995 alright. So that’s why the 0 data. Seems like I’m ok then!

[JulianCohen]

Looks good to me!

[Author]

Posts