Trading system broken?

[DanielBaeumler]

I’d like to get your view on a fundamental question: What are the criteria when you should stick to your system vs. when to stop/pause it because it might not deliver an edge anymore?

I’m currently struggling with my US MR system running on NDX. The system is pretty simple following 3 basic parameter. I launched it mid 2018. By then, backtest results looked like this.

Since then, I lost about 12% over the last 18 months. Backtest now delivers this:

I decided to switch it off (for now) mainly because the drop in performance looks rather ‘system inherent’ to me. On top, one of you recently posted in the forum that the number of professional traders running swing systems on NDX/SPX would have increased significantly over the last couple of years making it harder to generate a profit with MR systems.

Thanks for your thoughts.

[Nick Radge]

(1) How much optimisation did you do? I would consider a 69% win rate is reasonably high.

(2) Have your real time results more or less matched your backtest for the same period?

(3) WHY does the system make money? And has that condition changed.

[DanielBaeumler]

The system is based on relative strength and buys in oversold condition. Selling is happening either if market is overbought or after n days. If conditions have changed that this set-up is not delivering anymore money is the million Dollar question to be answered.
Optimization is only done with 2 parameters. Backtest results do not suggest any over-optimization (e.g. no performance peaks for any particular parameter combination).
Real-time results match backtest.

[JulianCohen]

One thing worth considering is that Larry Connors and Cesar Alvarez, the great proponents of RSI, are struggling with Mean Reversion systems at the moment. If you are using RSI or something similar to discover the overbought/oversold condition then that could be why.

Have you looked at jugging with that condition, to maybe find something else as an indicator?

[MichaelRodwell]

Has anyone ever seen a system results change so dramatically so quickly?

[ScottMcNab]

Daniel

At the risk of sounding like a broken record I will explain my approach to this problem. What is needed is a definitive test to allow you to decide if you will continue to trade this system. Stay with me as I digress for a minute and I will come back to the test I use.

I currently believe that the most likely problems I will encounter arise from either
1. curve fitting or
2. obtaining a good result (back test) purely by chance (having done over 15,000 back tests in the last year months this is highly likely).
There is also the possibility that
3. markets will change fundamentally in the future such that my current systems no longer work.

The only solution I could come up with to try and detect these problems and avoid them is using a technique that is applied initially in the testing phase and subsequently in the monitoring of the live performance of the system. My current solution is multiple markets. For me to continue testing any system, it must have similar expectancy (not cagr) in all markets or I throw it away. I have discarded several systems that were excellent for US or AUS markets but not both. Am I throwing out good systems? Highly likely, but I see it as the lesser of two evils. I would rather toss out a good system than risk trading one that has been curve fitted or obtained good results simply due to chance.

Performing this test on multiple markets in the design stage then allows me to also use this as a method of monitoring live systems. If the system performs poorly in one market but continues to perform well (paper trading) in the other markets then I will continue to trade it. It was a source of reassurance for me in 2019 when my US MRV system tanked in May that the system trading XAO did not. I also test and monitor my live systems on Tokyo (always enlightening), Korea, Hong Kong and London (problem with survivorship bias notwithstanding) but then I am particularly paranoid about losing my savings. If a live system performs poorly in one market but remains acceptable in others then I will continue to trade it.

In your case, I would start with NDX (always makes me nervous as such a small group of stocks with exposure to limited sectors), then move to RUI excluding NDX and then RUA excluding RUI (RUT)…again looking at expectancy (or win/loss rate, average profit/loss %, PF and payoff ratio. If you have ASX data then obviously that is good. I like to test on SPX back to 1960. To me, the extra (tax deductible) cost of the subscription is money well spent.

I have not been able to come up with a better practical test to make these decisions.

[JulianCohen]

This is the underwater equity for a system that I wrote and started to trade in early 2018. MOC system where I was basically looking for ROC of stock to be less than ROC of Index. Very few parameters and not optimised as far as I can tell.

I quit it when the drawdown was twice what it had been in the past ten years, at around 20%…This might not be an encouraging example but I have written many systems since that I am trading so there is always something else out there.

One thing I will do is spend next weekend applying Scott’s methods to my MOC systems. Damn good idea and I should have done it before.

I guess the message in the story is that if you have fallen out of love with your system, if you doubt it so much that it is painful to trade it, then that might be a good time to quit it. Or do as Len suggests and paper trade it until the system comes back over 200 day MA. In the long run that might not be the best way to trade it as far as CAGR, but if it allows you to keep going then why not?

Just some more food for thought. We are all different and what works for one will not necessarily work for another.

[JulianCohen]

Just as a test I ran the same system through the ASX to test Scott’s idea.

That would have saved me a few quid!!

[DanielBaeumler]

Thanks guys. Excellent advise.

I remember, the chapter in our course talking about robustness states that a robust system is supposed to work in all market. It’s sometimes that basics we forget over the excitement of a promising backtest result…
Following Scott’s backtest logic is delivering following results:
RUI ex NDX

Backtest results for RUA lookin similar.

Considering that I did not make any parameter optimization, backtest results look reasonable to me.
However, coming back to Julian’s point. The system is using a RSI related indicator which seem to be not a good choice these days. Hence, my approach will be as follows:

1) Get rid of RSI/RSI related indicator (which will probably require a complete redesign anyway…)

2) Backtesting following Scott’s approach

3) Focus on RUT (assuming that this market is less popular for the bigger ans possibly more professional system trader)

[DanielBaeumler]

.. in cluding the backtest results

[TrentRothall]

Good discussion guys.

Daniel one thing i do too especially in the US is try to test after the year 2000, this is mainly because MR system seemed to go crazy over the Dot Com Boom. I have found if you have a year with super high returns it can skew the final figures a fair bit, but then in saying that it probably isn’t the worst to test over that period because of the different market conditions.

For interest sake i ran my ASX MR on the rus1000, it isn’t pretty. Hopefully not a sign of things to come

[TrentRothall]

Also Daniel for that system you posted I see that your average win is held for approximately 6.7 bars where your average loss is nearly double this. I think I read somewhere that you have a N-bar exit, have you tried to reduce this down to 8 or nine bars? This might help reduce some of your drawdown also might help with trade frequency.

When I was building my system I did a back test with the N bar exit of something like 20 days I then dump the results into excel and sort then from bars held and calculate what are the chances of having a winning trade after x-days. It also helps psychologically instead of seeing the position going down down down down.

[Anonymous]

Daniel,

My two cents if you’re interested…..

1. In my opinion the equity curve you presented in the original post is just too good to be true. It doesn’t even have a blip for the 07/08/09 headache. Also it is interesting to see year 2000 as such a juicy year right from the first month. So I know you’ve said no curve fitting was done, and perhaps you landed on a unique set of parameters that produced this great result, but did you do the part of the testing process to DISPROVE the system/results/settings? That is, did you deliberately alter the three parameters you are using up/down/left/right away from the chosen settings to see how stable/robust they were at adjacent values? Did adjusting by just a little see the whole thing fall over? Or did not much at all change even if altering the values by +/- 20% ? If one of the parameters was set at 100 days, did it also provide good results/curve at 90? 110? 80? 120?

2. I have found that whenever I see such a wonderfully perfectly logarithmic curve like this that it is just too good to be true, and only upon further digging I have done something stupid like forgot to put a ref(X,-1) in the right place and I’m using the result of a number from the same day in which the test is performed whereas I should be using a number from the day before, and this post-dictive error then makes the result take off like a rocket and I start to think Iv’e found the holy grail. Of course, on further digging I find my mistake and my head comes back to earth. SO I would scour your code for the correct implementation of buysetup, all the ref(x,-1) items in the right place etc.

3. Was the system heavily weighted on Crypto boom related gains up to mid 2018? When I am building my systems and backtesting/optimising I deliberately keep Crypto boom related stocks on an exclude list to keep them out of the backtest. It is not a lot, just NVDA, MU, AMD and MSFT. This is because whilst I believe that other such booms could happen into the future, the optimisation/backtesting process gets so muddied up when they are included in the test with wanting to push you towards less and less positions with larger allocations per position, simply because it is seeing these big booming handful of stocks from 2015+ because of cryptomania. In real trade I don’t keep them in an exclude list, I still trade them if they come up on my explorations, so usually I will build systems whilst excluding them then once built and settings are done I remove them from the exclude list and usually my results will alter only very slightly upon their inclusion. If my results alter radically with their inclusion at this last step, then I know my weighting towards them is too heavy and I need to revisit the drawing board. If you are running an NDX system with a limited number of positions (let’s say 5 positions) then you could well be getting a lot of trades on NVDA and MU through to mid 2018. Since mid 2018, the crypto mania has receeded some, so perhaps this might be why its falling over post 2018. Is this so? Are you getting a lot of NVDA and MU trades in the backtest up to 2018?

I hope these ideas help.
Matt

[JulianCohen]

One other thing to try is to set the exit bars to 5 days or less (make money or get out), and I also set an exit if the stock falls below the MA that you have as part of the setup.

Good luck

[Anonymous]

I had another thought. I know it may be a silly one, but hey you never know….

Did you change any of your data subscriptions/packages around mid/end 2018? For example were you subscribed to a certain package of data, renewed during 2018 and even though you believe you may have renewed the same product annually maybe the data provider dropped a certain set of data from the package (without you knowing) like currencies. Perhaps the data was being updated/populated in the way you always knew up until that time, you renewed and believed everything was correct/same but actually a certain feature of the data is now missing compared to what you previously had. The old data will stay in place and only new data is updated beyond that point. If it something like currencies, maybe up until that point you had the currency data updated daily, but beyond that point currencies never got updated anymore, so your backtests in EURUSD were dynamic each day up until mid 2018 and it has been stuck on 1.20 exchange rate ever since because the currencies are not updating any more in your database.

Have you given the code to someone else to try on their installation of Amibroker?

Did you upgrade Amibroker around that time 2018?

Maybe starting from a fresh new installation of Amibroker, a completely new fresh/download of all the entire tick database from data provider and then trying again?

Is the in-memory cache size (max symbols) and Max Megabytes set correctly in settings? Even if you think it is set correctly, it is sometimes good to change it by just one digit (for example mine is set to 35,000 but sometimes I go in there and change it to 35,001 simply so that Amibroker confimrs the change and asks “you have changed data cache settings. Do you want to flush the quotation cache to apply the changes right now?”. I hit yes, simply to just get the cache flushed. I don’t know if the cache flushes every time you open/close Amibroker, but I do this every now and then to make sure I give backtests/optimisations the best chance at running quickly.

I know I’m grasping at straws now, but sometimes the silliest/simplest of things can be at fault. As long as you have your formulas, indicators, lists and anything else you want to save somewhere safe outside of Amibroker it doesn’t actually take that long to completely uninstall amibroker and norgate data, then reinstall and set it all up again. Sure, it might take an hour in total including waiting 20 minutes for all the historical database to download again, but an hour of farting around doing it is not that big of a problem in the grand scheme of things. I often find that doing things like this might force you to come across a setting that you need to set which you completely forgot about before and may also clean something up that you completely forgot about which you altered a year ago and always intended to change back again but completely forgot about since.

[Author]

Posts