Selection bias – how much is too much and general MOC discussion

[Stephen James]

Yes Scott,
PositionScore takes an abs value.
What are you trying to do exactly?

[ScottMcNab]

Trying to get a positionscore formula that would pick the worst trades possible…to see maximum impact of selection bias…

if system tended to have excess number of possible trades (ie selection bias) on days where the overall outcome was a market rebound then its not going to be the same issue as if the system was exhibiting selection bias on days the market tanked

kind of want to put the issue to bed (if possible)…I have been circling back to this issue periodically for twelve months.. i thought a positionscore that selected worse possible outcomes for the day might help….

anyone remember how positionscore deals with positive v’s negative values in long only system ? Can’t find an answer

[TrentRothall]

It will just take the highest numeral regardless if it’s positive or negative. ie -150 > 120.

If selection bias is only affecting <10 days a year as has been suggested somewhere is it a massive problem?? i mean nothings perfect. You'll probably miss that many a year due to mistakes, sickness, holidays etc anyway

[ScottMcNab]

Even though it is only on 5-10 days a year, the times it has happened in the last twelve months it has had a pretty significant impact..if I can get a formula for position score then I can use it in designing systems with minimal impact of SB with some metrics to give me confidence in it

Maybe something as simple as this then….(not sure if ok to use buyprice this way…?)

PositionScore = IIf(Close

[SaidBitar]

I do not believe you can make this with position score
…
OK it can be done but it is complicated
the easiest way to get the worst scenario make explore to get all the trades with the entry and exit price
throw them in excel check the days where you have more than the maximum allowed
set the starting capital and position size
for the days where you have more than the max allowed keep the worst and delete the best till you have the number you want
calculate the equity and plot it and then you will have the worst (i can make the excel tonight that will do this stuff automatically)

if you want to do it in position score
here is the way
you run scan or explore in order to store in an array all the trades with the dates and the profit
then you use position score = 1000 – profit

this case the biggest loser will have the highest score and the winners will have lower
and you will be taking the worst case everytime

[ScottMcNab]

Thanks Said…have I used buyprice incorrectly in the positionscore formula ? I was trying to make the number large on losing days (by adding the size of the loss to 1000…bigger the loss the larger the number) and small on the winning days (by subtracting the size of the win from 100)

[SaidBitar]

I believe I messed up the positionscore part i need to test it

[SaidBitar]

Scott
I believe your formula will work but no need for the if
Try this
Positionscore = 1000-(o-ref(limprice))

In this case if open is lower than the o-limit is negative and the result will be greater than 1000 and if the open is higher than the limit then the result is positive and positionscore is less than 1000

Positionscore will take the highest value in case of more trades than the allowed and I think this will give the worst case

[ScottMcNab]

The formula seems to work.Below is the chart showing days when buysignals outnumbered positions available:

none between 2012 to current
2011: 2 days
2010: 2 days
2009:none
2008:1 day

Below is an excel of CAGR from 1 jan to 31 dec for each year using the 2 different positionscores (random and then the one I have posted above)

with the effects of compounding, using positionscore = random on a system where are 5 days from 2008 to current when SB is an issue CAGR goes from 26.83 down to 20.6

I picked my asx system because I knew it had low SB…might go and look at US now

[ScottMcNab]

….and max DD (single run) went from -15.87 to -30.26

[ScottMcNab]

not happy Jan !

US MOC system been trading live:
backtest 2008 – current using positionscore = random: cagr 48.87% / maxDD 14.94%
backtest 2008 – current using SB positionscore : cagr 13.36% / maxDD -54.75%
Fail

Interestingly, the system I came up with after discussing on forum last week is ok…I was reluctant to switch as the CAGR was so much lower and I thought that SB probably wouldn’t matter..
backtest 2008 – current with positionscore = random: cagr 30.36% / maxDD 10.81%
backtest 2008 – current with SB positionscore: cagr 25.43% /maxDD -11.42%

[TrentRothall]

wow that’s confronting! Are you sure your code is correct?

[ScottMcNab]

I am never sure my code is correct.

PositionScore = IIf(Close

[RobGiles]

Hi Scott, Trent, Said, Julian, anyone else interested in MOC systems….

As you know I’ve only been trading mine for about 10 days and am yet to experience a situation where there are more possible fills than capital. Reading through the various posts on the forum on MOC systems and the issues around SB, am I right to assume that the MCS analysis and backtest results I did for my system are unlikely to be realised?  It seems that most people are getting significantly poorer performance than anticipated.  If so are we in the wrong market type to be trading such a system? Has anyone traded such a system for greater than 12 months and happy with it?

[JulianCohen]

Rob the way I see it is that selection bias, especially with a hi frequency MOC system is very important. If you are running a system with 90-95% selection bias and getting MCS backtest results of 40 odd% then very likely you can expect less in real time. I have structured my systems with this in mind. Running at 98.5% and showing 32% CAGR I will be quite happy to get over 30% for the CAGR in real time.

Remember that the MCS backtest only gives you a reasonable expectation of what you might achieve. The variation gives you an expectancy of how the results might change. It is always feasible that the actual results will be worse, it probably not by much. They could also be better…..

[Author]

Posts