Selection bias – how much is too much and general MOC discussion

[JulianCohen]

I just tested my systems with this Selection Bias code and there was no noticeable difference, which is what I would hope for as the selection bias is 98% for both systems.

From what I have found of testing for the past two months, my systems with 90% selection bias run at about 60% CAGR. They halve to 33% when I tighten the selection bias to 98%

[ScottMcNab]

Brent,interesting to find we have approached the same problem in similar ways. Calculating the worst case scenario is a personal preference that gives me peace of mind. After is has passed all the other tests, selection bias remains the elephant in the room with these systems. If I would still trade the system with the worst possible outcome from SB, then I can accept it and move on.

I think selection bias needs to get down to under 2%. In the asx test I posted above, there were only 5 days in 10 years where selection bias was in play and CAGR still dropped from 26 to 20. However, even these 2% or 5% figures don’t really accurately assess the impact for me and I think some form of postionscore provides more meaningful results for backtesting and system design.

Initially, MRV were the only systems I had and I went with MOC because I could not get my head around trying to calculate the impact of selection bias in systems holding positions for 3-5 days. Now, I have some rotational systems about to be added and I can see the value of a MRV MOC system for those years when rotational systems struggle. So I feel it is worth the continued effort. I think Julian only re-examined this topic in the last month or so and seems to have solved the issue with a new system designed from scratch.

Have you had a chance to listed to the recording from the last Mentor Chat ? Nick mentions success he had reducing selection bias by calculating 2 different buy limit prices and then selecting the lower buylimit price. It may be worth combining two of your systems this way?

On the US markets, I was unable to get selection bias to even close to acceptable levels without 40 positions at 10%. The systems I am tinkering with give a CAGR of 25-30 with maxDD of 12-20 depending on the time frame selected as the values used in the SB filters.

[RobGiles]

Julian Cohen wrote:

I just tested my systems with this Selection Bias code and there was no noticeable difference, which is what I would hope for as the selection bias is 98% for both systems.

From what I have found of testing for the past two months, my systems with 90% selection bias run at about 60% CAGR. They halve to 33% when I tighten the selection bias to 98%

If you end up getting close to 33% return in reality, you’d have to be happy with that right? Are you confident that’s what it’ll yield Julian?

[RobGiles]

Scott, are you saying that you are not comfortable leveraging up 4x? This is what I have elected to do in my system t get the SB down, so I’d be interested to understand why you have maxed out your leverage at 2x.

Maybe 40 positions @ 5%?

[JulianCohen]

Quote:

If you end up getting close to 33% return in reality, you’d have to be happy with that right? Are you confident that’s what it’ll yield Julian?

yes if I get 33% in reality I’ll be dancing! Watch this space

I’m as confident as I can be now. I have pushed this to the max and worked on it for over a year, with many many perfectly good systems tossed out along the way. I say perfectly good because at the time I was very happy with them, until I discovered and understood the effects of selection bias. Now I am much more rigorous in what I think is good.
If this doesn’t work I’m going back to my old job as a celebrity gynaecologist.

[ScottMcNab]

Rob Giles wrote:

Scott, are you saying that you are not comfortable leveraging up 4x? This is what I have elected to do in my system t get the SB down, so I’d be interested to understand why you have maxed out your leverage at 2x.

Maybe 40 positions @ 5%?

I might have confused things a bit there Rob…I use 40 positions at 10% (4x) as I needed this to reduce selection bias and achieve the return I was after

[SaidBitar]

Here is the excel that will generate the worst case scenario for a MOC system

it is made of two parts
the first part it should be used in Amibroker just add this code to your AFL and it will generate all the trades for any duration with the entry price and the exit price

Code:

Filter = BUY ;
AddColumn(C,”Close”,1.20);
AddColumn(LEPrice,”Entry Price”,1.2);

after you add this code you need to run explore on any date duration you want to test on. It will generate ticker,Date,Close,Entryprice

then open the attached excel file and go to controls tab hit “Clear Data” button to remove all the previous inputs and data

then copy the results of the explore from Amibroker and paste it in “input” sheet in cell A1

then back to controls sheet fill the required information for starting capital position size commission information the maximum allowed trades on any date and press the “Generate Data” button.

the file will go through all the days and check the number of trades on it and if any date has more than the maximum allowed then it will sort from the worst to the best and will start deleting the best trades and keeping the worst trades till the number of trades is equal to the maximum allowed. then it will generate the position size, number of shres, profit, calculate the ending capital and the DD

all the statistics will be filled in the “Stats” sheet and will generate two graphs one for the equity curve and another one for the DD

[JulianCohen]

That is brilliant!

You are an Excel wiz. I just ran my MOC 2007-2012 which is my bogey period and it still made money. Sure the drawdown was 16% which is worse than the 6-7% it does the rest of the time, but it still gave a 250% return over that period so I now feel much more confident about the system.

[ScottMcNab]

I just ran a quick test looking at results from your excel file verses the position score formula I was using (over a random 8 year period):

first figure is from Said’s exel/second is from Amibroker backtest using positionscore formula:

trades: 5407/5407
winning 2816/2805
losers 2591/2602
win% 52.1/51.88
av win 1.55%/1.53%
av loss -1.30%/-1.33%
win loss 1.19/1.20

I now have more confidence in using that positionscore formula for designing and testing a system with low SB.
Many thanks again Said

[SaidBitar]

you are welcome guys

most probably what i will say now will sound stupid but around 1 year before i did the same exercise to check the worst scenario for each system even when the SB was bad such as capturing 65% of the trades and i saw that the worst case is still Good. So i said why to remove the SB if the worst thing that can happen will still make money. and i saw that the edge per trade is around 0.2% this means to make more gains you need more trades and this is why i started trading lots of systems that are very similar in nature. Honestly it is a bit aggressive way of trading and it is a two edged sword since the DD of all the systems happen at the same time and you will find the account DD is getting deep very fast even faster than all the systems but the good edge of it when the market situation turns bullish then you can see big jumps in the final equity. I remember March 21st the US market was really bad and i went to fully loaded directly after the open and the account went down 10% in one night. 10% of the capital disappeared March 22nd the account went up 3.4% so you can imagine the types of swings that will happen in combining the two evils over trading and SB. most probably if the market conditions were good for all the year i will never think about removing the SB why to crawl when you can run but i think luckily i learned the lesson before i got burnt

Recently came back to my senses and decided to play safe and survive longer. So SB was removed.

[JulianCohen]

Scott McNab wrote:

I just ran a quick test looking at results from your excel file verses the position score formula I was using (over a random 8 year period):

first figure is from Said’s exel/second is from Amibroker backtest using positionscore formula:

trades: 5407/5407
winning 2816/2805
losers 2591/2602
win% 52.1/51.88
av win 1.55%/1.53%
av loss -1.30%/-1.33%
win loss 1.19/1.20

I now have more confidence in using that positionscore formula for designing and testing a system with low SB.
Many thanks again Said

Is there any reason the position score code will not work on an MR system too? I can’t see why not but just checking

[JulianCohen]

Said Bitar wrote:

you are welcome guys

most probably what i will say now will sound stupid but around 1 year before i did the same exercise to check the worst scenario for each system even when the SB was bad such as capturing 65% of the trades and i saw that the worst case is still Good. So i said why to remove the SB if the worst thing that can happen will still make money. and i saw that the edge per trade is around 0.2% this means to make more gains you need more trades and this is why i started trading lots of systems that are very similar in nature. Honestly it is a bit aggressive way of trading and it is a two edged sword since the DD of all the systems happen at the same time and you will find the account DD is getting deep very fast even faster than all the systems but the good edge of it when the market situation turns bullish then you can see big jumps in the final equity. I remember March 21st the US market was really bad and i went to fully loaded directly after the open and the account went down 10% in one night. 10% of the capital disappeared March 22nd the account went up 3.4% so you can imagine the types of swings that will happen in combining the two evils over trading and SB. most probably if the market conditions were good for all the year i will never think about removing the SB why to crawl when you can run but i think luckily i learned the lesson before i got burnt

Recently came back to my senses and decided to play safe and survive longer. So SB was removed.

What you could do now is to use one system with expanded selection bias as an ‘aggressive model’. So as long as you are happy with the worst case scenario situation, and selection bias bigger than your normal systems, the aggressive system will have bigger swings in P/L but hopefully better returns. Diversification at work.

[ScottMcNab]

Is there any reason the position score code will not work on an MR system too? I can’t see why not but just checking[/quote]

Not sure…maybe worth trialling in MR system over a month or two when there is NO selection bias and seeing if it at least gives the same results as positionscore = random….if it does that then can progress on to trying results when it some SB …t

[SaidBitar]

Julian Cohen wrote:

Said Bitar wrote:

you are welcome guys

most probably what i will say now will sound stupid but around 1 year before i did the same exercise to check the worst scenario for each system even when the SB was bad such as capturing 65% of the trades and i saw that the worst case is still Good. So i said why to remove the SB if the worst thing that can happen will still make money. and i saw that the edge per trade is around 0.2% this means to make more gains you need more trades and this is why i started trading lots of systems that are very similar in nature. Honestly it is a bit aggressive way of trading and it is a two edged sword since the DD of all the systems happen at the same time and you will find the account DD is getting deep very fast even faster than all the systems but the good edge of it when the market situation turns bullish then you can see big jumps in the final equity. I remember March 21st the US market was really bad and i went to fully loaded directly after the open and the account went down 10% in one night. 10% of the capital disappeared March 22nd the account went up 3.4% so you can imagine the types of swings that will happen in combining the two evils over trading and SB. most probably if the market conditions were good for all the year i will never think about removing the SB why to crawl when you can run but i think luckily i learned the lesson before i got burnt

Recently came back to my senses and decided to play safe and survive longer. So SB was removed.

What you could do now is to use one system with expanded selection bias as an ‘aggressive model’. So as long as you are happy with the worst case scenario situation, and selection bias bigger than your normal systems, the aggressive system will have bigger swings in P/L but hopefully better returns. Diversification at work.

For the moment I feel happy and relaxed knowing that I am on the correct track now it is waiting game and placing orders hopefully slowly and steady will reach my target

[LEONARDZIR]

Said,Scott,
I have been running your worst case scenario codes and have a question.
In my MOC I would like to code so when I backtest I can exclude days when I have more buysetups than number of positions allowed. Currently I can take 40 positions max. I would like to know results of my backtest when I exclude days with more than 40 buysetups.
The goal is maximize my returns on days when I have less buysetups than positions allowed. I believe Nick mentioned this approach on the last mentor call.
Thanks

[Author]

Posts