Selection bias – how much is too much and general MOC discussion

[SaidBitar]

regarding winning percentage my best is 57% i couldn’t reach even 58%
my profit factor is in the range of 1.3
i am capturing around 70% of the trades so i have selection bias i prefer to be capturing around 85% but the problem when i reach this target the returns are not interesting anymore
regarding the margin i start with 20 positions and 10% each and based on the results and DD i start increasing the margin or the number of positions for example if i have 15% CAR and 8% DD i will increase the margin and i am happy with 30% CAR and 20% DD
regarding the noise that Trent talked about i believe i have lots of noise i need to figure out how to filter them out without degrading the results
Decreasing selection bias here are what i did and where i reached
increasing the stretch is helpful to a certain limit end of the day you want to have trades and not only make it impossible to get filled
loosening and tightening the parameters help but you will notice degradation in CAR and sometimes DD will not decrease as much as CAR
decreasing the universe it is helpful like moving from Russell 1000 to S&P 500 but lower than this i believe it will decrease the frequency that i desire since my profit factor is small and winning percentage is not much better than the losing percentage i depend on trade frequency to balance stuff out

I started trading MOC after Brexit actually my first time was on Brexit so not enough knowledge about the best practices.
but what i can say maybe at the start the deviation between real time trading and backtesting will be huge (no so huge though) but with time they will start coinciding so even if you have selection bias you only had 1 week of real results keep it running and you will modify on the run all the time at least this is what i am doing

[LEONARDZIR]

My take. I have been trading an MOC system in the us market since August. If I trade my system with 4:1 leverage in the Russell 1000 I get a CAR of >60%. I started trading that way in August because I thought I had discovered the holy grail. I finally realized that my live testing would not match my backtesting because of selection bias. So I realized that selection bias is a really big deal.
Currently trading the sp500 at a reduced size with 4:1 leverage(40 positions at 10%) with CAR of 43% ,16% did over 10 years. Most importantly I now capture more than 90% of all trades. 56% winning trades, profit factor 1.3. With this system commission drag is very important. I use IB and commission costs are about 30% of profits.
i have also noted a drop off in results over the last 5 years but my system up 22% this year. Hopefully not a change in market structure that is permanent.
I might mention t

[LEONARDZIR]

I also trade a rotational momentum system that has a ver low correlation with my MOC system.

[ScottMcNab]

I may be doing it incorrectly but I do it with fixed to remove compounding

[TrentRothall]

Scott McNab wrote:

I may be doing it incorrectly but I do it with fixed to remove compounding

That’s correct if you want to know the daily profit/loss in dollar terms per $10,000 position for example. Where if you have compounding on you look at the figures in % terms.

[ScottMcNab]

Doesn’t compounding have the potential to skew the results a bit depending on which trades were taken (due to selection bias coincidentally) ? Probably not a big deal.

[TrentRothall]

Not exactly sure what you mean

[ScottMcNab]

Suspect I’ve muddled it up…haven’t given it much thought in a while

[Anonymous]

Thanks for all your feedback guys – big help.

I think I have something I can work with now. Depending on the leverage, I am getting CARs ~25% with DD of ~10% in the US and similar numbers in AU using the ASX300. This is based on leverage of 400% in the US and 200% in AU.

The numbers above in the US are based on 25 positions @ 16% each, and in AU are based on 15 positions at 13.3% each. Not sure if I will push it that hard in the US – will probably start lower and go from there.

Given the smaller universe in AU, I don’t see any way of running into selection bias – even with a lower base stretch factor in AU, tests show 100% signal capture.

In the US, tests show 4% of days with a trade having signals above my threshold (4% of days with a trade have more than 25 signals, opening the door for selection bias). I am comfortable with 4%, but if this ends up being too high I can manipulate the parameters to bring this down.

I think signal capture % is a misleading metric and doesn’t accurately measure selection bias – I think it overstates it. Regarding the above in the US, my signal capture is only 80%, but the majority of the missed signals come from days when there were 100+ signals (signal clustering), so gauging the selection bias on a “% of days with selection bias” seems to make more sense.

On another note, I’ll share a few things I did to try and decrease selection bias while maintaining returns. I’ll note that some did indeed reduce selection bias, but all eroded returns to the point of the system not being interesting (decrease N), so these points are more about me sharing what I tried that didn’t work than anything. Keep in mind this is for a simple reverse channel system.

1 – Used an individual stock ROC filter: ROC(x) > n%
2 – Used an individual stock volatility filter: ATR(short) > ATR(long)
3 – Used an individual stock volume filter: volume past (short) days is elevated relative to volume past (long) days
4 – VIX filter – turn the system off if volatility is high or low relative to history
5 – Index filter – turn the system off if the broad market is in an uptrend or downtrend defined by the index level MA
6 – I did a lot of work on adjusting the ATR stretch multiple (ATRM) based on the VIX. I expressed the VIX’s current level relative to the highest high for the VIX over the past N days, and I developed a number of schemes whereby I would define the ATRM based on where the current level of volatility sat relative to the lookback period. This type of analysis almost completely eliminated selection bias, but cut the returns in half, making them uninteresting. I ran four different versions of this metric but struck out on each.

[LEONARDZIR]

Brett, not sure how you arrived at 25 positions at 16% but I think you will find in live trading that anything more than 10% per position will be too volatile. I settled at 40 positions at 10% also allowing me to trade more signals.

[Anonymous]

Thanks Len. I arrived at that sizing because I was trying to balance utilizing capital to the fullest extent without introducing significant selection bias.

When I run an exploration over the test period, I find that with 25 positions, no more than 4% of days with a trade will have any type of selection bias. This seems reasonable to me. So I ran the simulation with different levels of leverage and size allocations. The MAR profile across all sizing schemes is similar, leaving the only question of how much DD am I willing to take in absolute terms.

Under the 16% position sizing, the max DD over MCS (1k runs) averages -10%, with a max DD out of all the MCS showing -12%. A 12% DD is perfectly acceptable, so in my view, it makes sense to trade the leverage given I can withstand that drawdown.

I am curious – are you saying that, despite what the 1K run MCS is saying, that I should expect even worse DD in live trading? If so, why? I know a very successful and well seasoned trader who likes to say “your worst drawdown is always ahead of you” and there is something to that. But even if I add 50% to my worst system drawdown I am still sitting at -20% DD which, given the returns and my risk tolerance, is fine. So based on the testing and my risk tolerance, it seems it would be irrational not to allocate 16%/position. That is unless there is some reason to expect that real life will not mirror testing results in any way…but then what is the point of testing and trading systematically at all if we cannot trust our test results?

[Anonymous]

I think the discussion about selection bias and how many positions to use is really interesting when you get down into the nitty gritty of it.

Regarding the system I discuss above (Russell 1k, 25 positions), 96% of days with a trade will have 25 or fewer trades. Based on the distribution of the number of trades per day, I could see someone arguing to use an even lower number for the max positions, but then the position size would really get out of hand (16% as outlined above is a bit high for my comfort…not sure if I will actually deploy at that big of a number).

The below data shows the frequency distribution of valid signals for the last 5 years. Most days with a trade see 1-4 signals trigger, with the likelihood of more signals triggering falling as the trigger number goes higher…nothing surprising here, its just interesting to see that when employing this system, the chance that you will hit your max number of positions threshold is extremely low.

I am aiming to keep the number of days where selection bias will occur at <5% of all days with a trade. I am not sure if this is the correct number, but then again, I am not sure there is a way to tell without deploying live.

I suppose one could look at the days with selection bias individually and run some sort of test to get the worst possible outcome possible on those days and then “adjust” their test results to reflect that. Kind of wonky but it may give more insight into how big of a problem selection bias in a system “could” be.

[Anonymous]

Just doing a quick run on the above system, I find that increasing the number of signals and decreasing the allocation actually ends up increasing selection bias on a like-for-like return basis.

Iteration 1:
25 positions @ 16%
Stretch factor 1.5
CAR 27%, Max DD 10%, win rate 56%, days over signal threshold 4%

Iteration 2:
40 positions @ 10%
Stretch factor 1.5
CAR 15%, Max DD 8%, win rate 55%, days over signal threshold 1%

Now decreasing stretch on iteration 2 to get a like-for-like comparison on a CAR basis to iteration 1:
40 positions @ 10%
Stretch factor 1
CAR 26%, Max DD 11%, win rate 54%, days over signal threshold 8%

[LEONARDZIR]

Brett,
Yes expect greater drawdowns than backtesting especially if there is some selection bias but even without. I traded a 15% per position for a brief period In small 50k account which was ok but when I started trading with a substantially larger account I found it too volatile. Nick suggested the 10% figure to me.

[ScottMcNab]

Any thoughts on using the following to determine the impact selection bias will have on your MOC system much appreciated…

PositionScore = 1/(Close-Low);

Look at drop in performance verses when use
PositionScore = mtRandom();

[Author]

Posts