Selection Bias

[ScottMcNab]

My MOC system tracks backtest except for days when there are large numbers of orders. On those days,if there are many more orders than positions available, then bets are off and the distribution of returns becomes uncertain …I have had a two occasions where I have done a 1000 run MC over the day in question (over 100 orders were generated but only 25 positions) and my actual fills were just about the worst possible…hah…may just have been chance but then again it may be that the first 25 orders that trigger at the open are the ones tanking heavily…would need intra-day data to know of course.

These days have had an impact on creating a difference in the actual live system return for the month involved compared to the backtest in the order of 1%..which would hurt over a year…..so I am in the process of testing tighter requirements and/or increasing number of positions to try and minimise this…dont think can ever eliminate it…. Only been trading MOC system for 3-4 months live though so in the end may all even out but I don’t enjoy the uncertainty generated by this phenomenon.

[ScottMcNab]

My MOC system tracks backtest except for days when there are large numbers of orders. On those days,if there are many more orders than positions available, then bets are off and the distribution of returns becomes uncertain …I have had a two occasions where I have done a 1000 run MC over the day in question (over 100 orders were generated but only 25 positions) and my actual fills were just about the worst possible…hah…may just have been chance but then again it may be that the first 25 orders that trigger at the open are the ones tanking heavily…would need intra-day data to know of course.

These days have had an impact on creating a difference in the actual live system return for the month involved compared to the backtest in the order of 1%..which would hurt over a year…..so I am in the process of testing tighter requirements and/or increasing number of positions to try and minimise this…dont think can ever eliminate it…. Only been trading MOC system for 3-4 months live though so in the end may all even out but I don’t enjoy the uncertainty generated by this phenomenon.

[Nick Radge]

This is a very important issue and its popped its head up a few times in recent weeks.

The simple answer is defining what is signal, i.e. the edge your system has, and noise.

On big days your system is catching a lot of noise and not much signal and is why your results are, and always will be, at the lower end of the distribution.

It may require I do a short little video to explain better, but essentially a MOC system will make profits when the close is higher than the open or the stretch is wide enough that it produces an edge.

On a busy day you will probably find you’re getting all your fills very quickly near the open and unless that open is a significant gap down the probabilities of the close being higher is somewhat less. In other words on a busy day you’re more prone to capture noise rather than those trades that provide the wanted edge.

Solutions:
(1) Increase the stretch
(2) Decrease the universe

Obviously this will detract from performance, HOWEVER, that performance will now be more significant.

[ScottMcNab]

Hope these images help. ASX uses 10 positions at 10%…..SPX is 25 positions at 8%…easy to see the day in Sep that caused me the grief in SPX

[SaidBitar]

I tried one exercise last month ended up no where but maybe you will have better luck than me.
the idea was to check the number of signals, number of trades, and the ROC of the index for the previous day.
I was trying to find a pattern so based on it I will modify the number of positions and the stretch.

[LEONARDZIR]

Scott,
Could you give us a little explanation for what we are seeing on the chart? What are differences between the upper and lower plots and what is the horizontal line on the bottom plots.
Also on your system what percentage of trades are you capturing?

[ScottMcNab]

Apologies….chart is borrowed from Darryl and is a plot of the number of buy signals generated each day for the MOC system. The horizontal line on the bottom graph is set at 25 which is the number of positions available for SPX.

I find this graph very helpful as I may have a backtest that shows exposure of 30% in one system and 20% in another but when I plot the buy signals there may still be a similar number of days where there is over double the number of buy signals relative to available positions. I am currently trying to keep the number of these days to once a month.

[Author]

Posts