Selection bias – how much is too much and general MOC discussion

This topic is empty.

Viewing 15 posts - 1 through 15 (of 125 total)

1 2 3 … 7 8 9 →

Author

Posts
November 14, 2016 at 8:54 pm #101588

Anonymous
Inactive

Curious to get the group’s opinion on this – particularly those of you who have been trading MOC systems live for a good while now…

I have been working on various MOC systems for a couple months now and I am having trouble coming up with something I feel confident in.

My first MOC system showed tremendous promise in the form of fantastic test results. I should have guessed that the results were too good to be true, but I gave it a try in live markets with a small account to see what would happen. The selection bias ended up being too high, and I consistently got results that were at the lower end of the MCS range. After 4+ days live, the actual results would be completely beyond the lower end of the MCS range – reality would not match the live simulation results. I tried three different iterations of live trading to see if I could fix it, but the selection bias could not be overcome in that system. With Nick’s guidance, I decided that I needed to develop a new system that would limit signals significantly and thus reduce selection bias.
So I went to work on a variety of different MOC models and found a fairly simple short term reverse-channel entry with a longer term stock specific MA filter combination that gave solid results. The testing shows a much smaller potential for selection bias, but nonetheless, the potential for selection bias is still there. Thus I remain hesitant to start trading again given my previous experience, where selection bias destroyed the system in real time – I feel like I can’t trust my test results given how poorly reality matched test results last time due to selection bias.

Some results / general commentary below. I have tried various index level and stock specific filters, but at least on this base rule set, rule/filter additions seem to detract from performance without any meaningful improvement in signal capture/decrease in selection bias. When evaluating the possibility of achieving results in real-time that are consistent with real time simulation results, signal capture, and “Days w trades over threshold” are the key metrics. “Days w trades over threshold” expresses the number of days where more than the max number of positions would have been possible, as a percent relative to the total number of days with a trade.

I am curious what others have seen in terms of selection bias in testing and how that correlated to real time performance. What levels of selection bias have you all seen and have you found a “threshold” level for signal capture below which selection bias begins to meaningfully impact real time results relative to real time simulations?
Also, what types of returns/drawdowns profiles are you all seeing in testing for MOC systems over this time period? Are my results typical of MOC systems that you all have tested and traded or should I get back to the drawing board and work on something altogether different?

Thanks for any and all input.

All below are based on 50% LTV.

November 15, 2016 at 12:05 am #105772

ScottMcNab
Participant

In no particular order my thoughts:

(as Said mentioned recently) take the results from MC (I use 1000 run over last 5 years at 20% trade skip) and take worst DD and lowest CAR…I then expect this…if it is acceptable then trade system

if the same system (with only changes to price/volume/turnover filters) doesn’t produce same backtest results (expectancy…not CAR) on ASX as well as US then I bin it (possible curve fitted)

rough rule…I keep exposure under 30% for MOC intra-day system…better still use Darryls graph…only want 1-2 days a month where buy signals greater than threshold…can scan several years worth of backtest in seconds with the graph and see clumpings (about 5% in your metric if 20-22 trading days a month so that seems ok in your R1000 results with 25 positions)

I switched from R1000 to S&P500 to reduce universe…currently using 25 positions in SPX but will go up to 40 soon (others use R1000 so this my thoughts only of course)

want profit factor above 1.5 (Nick’s recommendation/post)

my max trade drawdown also about 30%

58% win rate very similar

ave profit/loss of 0.3% better than my system gives on US market so I would not abandon system…maybe just take it to SP500, use 25-40 positions and see if need to tighten/loosen entry requirements …then take it over to ASX and see if win/loss rate and av profit/loss similar

I paper traded my systems for a while to get confidence (and in several cases found mistakes) to get the confidence to go live…if you bored read my journal…hah…should get your confidence back when read mistakes

November 15, 2016 at 12:51 am #105785

ScottMcNab
Participant

Brent…I should have added one more line…only been trading MOC live since Aug so please take all those comments simply as ramblings of an inexperienced trader

November 15, 2016 at 5:55 am #105787

LeeDanello
Participant

Brent, Looking at your stats, these are my personal observations:
1.Why aren’t you using 4:1 leverage? I thought that was the point of MOC sysytems.
2.Your Max Equity Drawdown is great. I aim for the low teens.
3. Your Profit Factor is low. There doesn’t seem to be much leeway between profits and losses. I aim for 2.0 and above
4. Your trade win is low, 70% is my cut off. Look at where you are on the curve – Ave Win/Loss v % Win and see where you are on the curve. I wouldn’t think you’d be that far right of the curve.

5 Your pay off ratio is good.
6. The variance between your CAR is a bit great for me. When you say iterations are these 3 runs only?
If so what about Monte Carlo?

Seems like it wouldn’t take that much to turn those winners into losers.

November 15, 2016 at 6:46 am #105786

Anonymous
Inactive

Thanks for your feedback Scott.

“(as Said mentioned recently) take the results from MC (I use 1000 run over last 5 years at 20% trade skip) and take worst DD and lowest CAR…I then expect this…if it is acceptable then trade system”

-This makes sense, and this is what I was going on when I ran my last system live. However, it is important to realize that even if the results themselves look good, they can be a red herring if selection bias makes them unattainable.

“if the same system (with only changes to price/volume/turnover filters) doesn’t produce same backtest results (expectancy…not CAR) on ASX as well as US then I bin it (possible curve fitted)”

-The results on the ASX are better, with less potential for selection bias. This is encouraging and I may end up running this system exclusively in AU rather than in the US.

“rough rule…I keep exposure under 30% for MOC intra-day system…better still use Darryls graph…only want 1-2 days a month where buy signals greater than threshold…can scan several years worth of backtest in seconds with the graph and see clumpings (about 5% in your metric if 20-22 trading days a month so that seems ok in your R1000 results with 25 positions)”

-I like this idea of targeting systems that will “clump” only 5% of the time. I have found in my testing going back that clumping has increased over time. I think this is part of a theme of increasing correlations over time.

“I switched from R1000 to S&P500 to reduce universe…currently using 25 positions in SPX but will go up to 40 soon (others use R1000 so this my thoughts only of course)”

-I may end up focusing on a smaller universe after all. The results just aren’t that great when you cut the universe in half. That tradeoff of more signals and better results vs more signals and higher potential for selection bias is tough to shake.

Thanks for your feedback Scott – I appreciate it!

November 15, 2016 at 6:57 am #105788

Anonymous
Inactive

Thanks Maurice.

1. I will probably end up using 4:1 leverage at some point…this is just a starting point.

2. Agreed – the drawdown side is very good. And if results increased with more leverage, I would be very comfortable with bigger drawdowns.

3. A PF of 2.0 seems pretty high. Are you seeing PFs of 2.0+ for MOC strategies?

4. Same as above…I haven’t found anything in researching MOC with a win rate higher than the low 60% range. Maybe I need to go back to the drawing board if 70% is achievable…

6. The “iterations” are different versions of the same system (different position sizing scenarios and/or different filters), but all essentially the same system. The results I posted are the averages from MCS under different filtering/position sizing scenarios.

Agreed that it wouldn’t take much to turn the winners into losers. And this is troubling. More troubling is the fact that performance (both on an absolute and risk-adjusted basis) is on a steady downward trajectory over the course of 2000-2016 (I restart the account balance at 100k every 5 years when I do extended testing to avoid the problem of the account getting too large to take meaningful position sizes, so this isn’t the issue here). Further, walk forward optimization doesn’t seem to help this, so it looks like the market is gradually shifting character against this type of system.

It looks like I may end up scrapping this and going back to the drawing board.

Thanks for your feedback Maurice.

November 15, 2016 at 7:24 am #105789

LeeDanello
Participant

Brent, The system I designed isn’t a MOC system. Not sure I could handle one. I designed it for the ASX All Ords and has a holding time of 5 days for winning positions and 8 days for losing positions. Based on MC, trade wins are 72.12% ave, 69.83% mean and 73.97% max. My profit factor has a min of 2 but my minimum payoff ratio goes down to 0.8 and a max of 1.0. I need the higher profit factor to make it worthwhile.

November 15, 2016 at 7:31 am #105790

TrentRothall
Participant

Hi Brent

Have you tried extending the “stretch” to minimise selection bias? This will probably help enlarge the average win size too.

I haven’t done extensive moc testing as yet, but i haven’t got anywhere near a 70% win rate either..

November 15, 2016 at 7:54 am #105791

Anonymous
Inactive

I’ve played with the stretch mostly with no real improvement SO FAR (that said, see next paragraph). The problem with extending the stretch without making a counteracting adjustment is the same as reducing the universe – you get smaller selection bias and smaller metric deviation, but at the cost of absolute performance since you limit the number of trades you take – same as if the casino shuts its tables down early – fewer opportunities to exploit its edge.

However, just tonight I am playing with a more adaptive stretch and channel lookback period manipulation and I am finding some interesting results. I will post about them later on.

Keep in mind that channel length and stretch are essentially two sides of the same coin:

-increase stretch, hold channel length constant –> trade frequency lower
-decrease stretch, hold channel length constant –> trade frequency higher
-hold stretch constant, increase channel length –> trade frequency lower
-hold stretch constant, decrease channel length –> trade frequency higher

70% win rate for MOC seems high – I would be interested to hear if anyone is seeing that type of result.

November 15, 2016 at 9:16 am #105792

LeeDanello
Participant

Brent, Just to let you know that if I test my system on the Russell 1K my profit factor drops to about 1.7. The metrics on my system are better on the ASX. Win percentage is just shy of 70%

November 15, 2016 at 9:17 am #105793

TrentRothall
Participant

I’ve found that taking less trades can be beneficial if you are eliminating trades that have little or no edge. The hard thing is finding the trades that are not doing what you expect, out of a back test with 1000’s of trades. That’s where adjusting the stretch (or a similar tweak) can come in handy. You are better off doing less quality trades than – like Nick said the other day, taking trades which are just noise.

So there might be something with a index filter or MA that may help, even though you have probably tried a lot of different things by the sounds!

November 15, 2016 at 9:19 am #105773

JulianCohen
Participant

Brent maybe try 300% margin…my SPX MOC is running at 30 positions at 10%

It may make the difference to the metrics you are looking for without fully maxing the margin

November 15, 2016 at 9:34 am #105795

LeeDanello
Participant

Nick Radge wrote:

A metric that I find very useful and rarely used is Profit Factor. Profit Factor measures the difficulty to trade the strategy. Anything under 1.5 in my view is probably a very difficult ask for most traders.

So 1.5 is a minimum, maybe 2.0 is ambitious

November 15, 2016 at 10:04 am #105774

ScottMcNab
Participant

I would recommend a walk forward test to see what the worst 3 or 4 daily drawdowns are (remember to change to fixed position size) over several years before I increased leverage to 300 or 400%… at 200% the worst days for each year are quite confronting for my tolerance levels

I have never come close to 70% for an intraday MOC system

The other factor for a MOC system (which have max DD in single figures) Brent is that it can be traded around the clock…so use the same money to trade both the US and AUS…this allows you to tighten criteria and accept the reduced returns (and reduced DD)…but if you are achieving 12% CAR on AUS and 18 % on US (with maxDD of 5-8% on each) then thats not too shabby in my book

November 15, 2016 at 11:59 am #105796

LeeDanello
Participant

Scott McNab wrote:

I would recommend a walk forward test to see what the worst 3 or 4 daily drawdowns are (remember to change to fixed position size) over several years

Why change to fixed position size?
Author

Posts