Matthew O’Keefe’s Journal

[Anonymous]

Scott,

I am sure that what you have posted from the Amibroker forum is true. However keep in mind I am only talking about the order of trades opening and closing within the one and the same bar of trade.

As per the post from Brakkar, we don’t know if they are asking the question for two potential trades occuring at two different dates or also within the one bar.

From how I understand it works, randomisation of trade sequence can affect the outcome for when you calculate the position size dollars of a second trade given what has happened to the equity position after the calculation of the first trade. I don’t want this to occur because the position sizes, in dollars, should be calculated all upon the equity of the previous bar, then traded all within the next bar, and the total new equity position is known for calculation for the next 23 orders that happen to be executing on the bar after that. Again, those 23 next trades on the next bar can occur in a randomised order, so the ending equity position of that next bar could well be different in however sequence those 23 trades fall within that day, however the dollar size position of each of those 23 trades would not change their potential dollar order size based on that position, all 23 will be calculated however large each order should be, and then just the sequence in which they fall is randomised within the bar.

[Anonymous]

Said,

I believe it is because of the custom backtester code. If you happen to be using the custom ranking code shared with us by Julian this code is going to find the top most trades every single time. Ever since I started running the monte carlo simulations via code within my AFL I would run ten thousand monte carlos and the same results would pop out every single time. This led me to believe the monte carlo simulation process was not effective on any system running this custom backtester code. That doesn’t mean it is or isn’t, this is just my experience and what I have found.

And of course I could very easily be 100% wrong. I too am learning and trying to find my way. Perhaps I have completely misinterpreted the use of the Monte Carlo process and this last page of the report.

What I do know is that if I take a very rudimentary view towards anything I do, if the report is telling me I am approaching a potential ruin, I listen to that and do something about it. I change the way I have designed my system, or I increase the number of positions I trade. After having read a few books in the last few months it is clear one of my previous downfalls was expecting/chasing profits that were too high (through less positions and more exposure per position). Now that I am trying to adopt a risk averse approach, anything which shows me potential problems with my system, I listen to.

This Monte Carlo page in the backtest report is a good example. If I set the setting to “Simulate using portfolio equity changes” then I always invariably get something like this:

I wouldn’t say that this version of the report is “useless” to me, however it doesn’t really help me much in building robust system design. If this is how the report usually looks on this page, I largely ignore it because the Max Drawdown numbers are all within a reasonable range and the annual returns too. So looking at it in this way doesn’t really help me too much with my decision making.

If instead I use the report in “trade list” mode, and I see something like this……

well then I know something is wrong and something needs to be done. Usually my leverage is too high, or my number of positions too low. Again, it comes back to risk management and over exposing myself to any one single trade.

Also keep in mind that we do actually have a trade list whenever we use the custom backtester code because that is what it is there for, to give us the list, ranked by our ranking measure, of what we are going to trade up to the maximum positions we have designed in our system. For an 80 position system, Amibroker is not going to take any of the potential trades from rank 81 to rank 253 (or whatever amount of potential trades turned up in the exploration that day) for use in the Monte Carlo. That said, if the trade list of the top ranked positions is used, when trades are “skipped” in the monte carlo test, some other trade within those top 80 trades in the actual trade list will be duplicated and used in that skipped position. It will not fill the skipped position with a trade from outside the top 80 trade list, it will fill it with a repeated/duplicated trade from within the top 80.

[JulianCohen]

Matthew O’Keefe wrote:

Said,

I believe it is because of the custom backtester code. If you happen to be using the custom ranking code shared with us by Julian this code is going to find the top most trades every single time. Ever since I started running the monte carlo simulations via code within my AFL I would run ten thousand monte carlos and the same results would pop out every single time. This led me to believe the monte carlo simulation process was not effective on any system running this custom backtester code. That doesn’t mean it is or isn’t, this is just my experience and what I have found.
.

This is exactly what should happen with the CBT code as the coded MCS from the course is designed to search for selection bias and skip trades. With the CBT ranking code there is no selection bias as the same trades will always be chosen. I actually never did that test but it’s good to know that that is the result.

I use parameter variance on my stress testing now, to make sure that altering the parameters by 25-30% still gives me an outcome that I can live with.

[ScottMcNab]

Thanks Matthew. I cannot say it is clear to me at this time but I am very grateful for the time you have taken explaining it. I will give it some more thought.

[Anonymous]

Scott, I don’t give my comment from a point of being an expert, because I am far from that. Just giving my perspective and experience as I have seen it. This is not advice or telling you what to do, or that I aam correct. I just hope that by sharing my journey it might help others to avoid pitfalls. Also it might also open up myself to others proving my perspective is incorrect, which means I will learn something too.

[Anonymous]

Julian, I also do the same. Try to ensure a nice wide band of parameter range can be had and still enable me to make money with respectable DD.

Also, I really love it when I might have say, 3 or 4 parameters in my system and I run a CMAE optimisation and across all the possible ranges of parameters, even within the first 20 or 30 tests, even with the most whacky goof ball settings that you always see in the extremes of the possible parameter settings in the first handful of tests, settings that you would just never actually use in real trade as they are at the extremes of the ranges, that even those idiotic settings will still produce a profit. Sure, perhaps the CAR is only 4% and the DD is 40% on those dumb settings, however the fact remains that if you traded the system with complete idiot dart at a dartboard settings you would still make money in the long term.

I am big enough to admit my systems were not like that in the beginning of my journey. I would have 3 commonly used indicators working together, stacked on top of each other, and then I would run the CMAE optimisation and check my computer 3 days later after it has done 30,000 tests and get a full-on robot chubby because it was showing a 40%CAR on 16%DD. There would be hundreds maybe even thousands of tests in those cases with negative CAR and massive DD, but I thought I knew better and thought “that’s what the software is for”.

I don’t fall into this trap any more. Now I build in a way that CMAE optimisations, on no more than 3 or 4 or maybe even 5 parameters need to be showing immediate and reasonable results in the first handful of tests, I mean, even after just 20 or 30 tests it needs to be starting to move in the right direction with reasonable results. Certainly after maybe one or two hours of testing, maybe 100 to 200 tests have completed, that the tests are showing tradable results. If it isn’t, I know there is an issue with my design and I stop the test and go back to the drawing board.

Having parameters as broad as a barn door and can still produce ok results is what it is all about. I know Nick tells us this right from the outset, however as a new trader it is very tempting to try and find the holy grail when you have great software at your fingertips.

And don’t get me wrong, if I can find bands of good parameters in a range, I will still target where the peaks are within that band. I would be an idiot not too. I’m talking if my ATR setting has a range of 2 to 300 and even the dumbest extreme settings like 2 or 300 are producing 4% CAR on 45%DD but around 80 to 150 are consistently producing anywhere from about 15 to 20% CAR and 10 to 15% DD then of course I will concentrate in that 80 to 150 zone and may end up picking a setting of 123 because that is the maximum CAR of 21% and minimum DD of 12%, however I am very comfortable in doing so, and I know that even if things change in the market I’m still probably going to hit a good 15 to 20% CAR or so anyway.

If I want massive profits, like 30% or more CAR per year, I know there are no shortcuts and I’m going to have to do something else like scalping, day trading, options etc.

[Anonymous]

Having a break from working on Monthly Rotation, and this is certainly interesting, and challenging.

On the point of MCS, my reading and interpretation of what is on the AB guide, is that by using “Don’t Change”, means that the MCS will go into the Backtest results just produced, and from those results simply rebuild a new equity curve, using the same $ / position sizes from that original Backtest, but in a random sequence.

If compounding in the backtest, then you may by the end of the backtest period have losing trades of $20,000 depending on;

– how well the system performed to that point
– period of backtest i.e. 10 years vs say 3 years.

In the following randomisation by MCS, a few of those losses can get thrown to the beginning of an MCS run, so quickly producing the ruin scenario.

With the system that you have shown Matt, and assuming compounded and using “Don’t Change” in MCS, to have no ruin situation whilst reaching the equity levels you do, is really quite something.

I have run a few a couple of tests on an MOC (in progress), using “Don’t Change”, starting the backtest at 3 years length, then increasing 1 year at a time, to see how quickly the ruin situation came in;

3 years – 4,000 trades OK and then up to
8 years – 10,000 trades OK
9 years – 11,000 trades RUIN hit

Equity not getting anywhere near what you have demonstrated.

But really helps make me consider the “Ruin” issue from an additional angle.

Thanks.

[Anonymous]

Michael,

I am sure my methodology is about as accurate as using a baseball bat to slice a pizza, but it gets the job done.

And yes, you could say how long is a piece of string because 3 year test no ruin, 9 year test no ruin, then 10 year test suddenly the 1% is ruin. So is 10 years enough? is 15?

There is certainly no black and white in the process. I for example, am starting the test at beginning 2003. Why? Well its more than 15 years and includes the 07/08 debacle. If it can survive through that, I’m happy. And yes, mine too if it can survive 15 years without ruin and then drop it down to just 3 years it is nowhere near ruin, so then the question again becomes am I worrying over nothing?

The laws of large numbers just get better and better and more accurate with extra volume. 500 is certainly better than 50. 1000 is certainly better than 100, and as you have seen in your example from just 10,000 to 11,000 shows a difference. Still, on the difference between ruin and no ruin from 10,000 to 11,000 I wouldn’t worry too much, except to say what was the 1% percentage of ruin at 10,000? I am guessing DD was already at something like 70 or 80% in that 10,000 run example. This is why I try to always get, even on a good 15 year test or 10,000 trade sample, the 1% DD to no mor than about 50 or 60%. No I can’t quite get there every time, but again, it is not a hard and fast rule. All I know is that in the early days I was absolutely getting charts like the one I posted above with the first four lines of the monte carlo page hitting Ruin and I just ignored it because I thought “Well I’m using Julian’s awesome ranking code, so I don’t need to worry about that”. I think that I was a little naive in that way, thus why I was always chasing returns like 45% and DD less than 20 across maybe 30 or 40 positions and then end up in frustration in real trade because it hardly matched backtested expectation.

And again, I cannot emphasize this enough, I am also learning and finding my way. I am not an expert and I am often wrong. I am happy to learn if I am not using this monte carlo page/results/settings properly or if I am interpreting the results incorrectly and need to be shown a better way.

[Anonymous]

Thanks for the feedback Matt.

I am zero expert on MCS – just trying to understand how you are using it, and learning as I go.

My 11,000 test where I hit ruin, the 1% Percentile had a max DD% of 65%.

I had the red line though flatline at 0, after what look like less than 500 trades.

Does the red line show the single worst run in the whole MCS?

I am assuming that even though the 1% doesn’t show ruin, within that 1% there could be at least 1 run, possibly more, that did hit the ruin level. (NB I used 5,000 as Number of Runs).

Thanks again.

[Anonymous]

From my understanding, yes, the red line represents the worst possible sequencing of the worst possible trades.

As per the help file in Amibroker, it states that due to trade skipping it would duplicate other trades to make up the quantity of fills for that day. I assume this means that if for example you are trading 50 positions, and on today’s trades it is going to have a full 50 trades, that some of the 50 MAY be skipped. Let’s say 10 are skipped, then those 10 would need to be made up from the other 40 trades, duplicates of them, to fill the 10 missing trades. This could mean that theoretically perhaps the very worst possible trade of the 50 is chosen and repeated 50 times. I am sure this would be close to statistically impossible/unlikely, but it is probably possible from a programming point of view.

I am sure the chance of all this happening on the worst possible sequence of the worst possible trades is a very doomsday way of looking at things, testing etc, however I have become quite cautious in risk aversion with building my systems now. I guess you could call it building robustness too. Perhaps following the red line an hitting ruin is only a 1% chance of happening, but still, I like to try and avoid it if I can. I certainly wouldn’t want to run a system that had say three of four lines, like right up to the 25th Percentile, showing as ruin. My systems did have that before, so it is no wonder I was always fighting with my emotions on why my real time trade wasn’t matching backtests.

[SaidBitar]

i was reading the documentation about monte carlo in amibroker and the way it does it is by shuffling the trades. So maybe it will take a trade from 2020 where the position size is 10x the initial capital and will place it in the beginning (2005) and this is why it is showing that DD will be 100%. Another thing in the documentation they mentioned that if there are over lapping positions then you need to use the portfolio not the trade list settings.
So when i switched from trade list to portfolio risk of ruin disappeared.

the reason this made me worried is because all of my systems including the rotational ones

here is the result of the DD for the rotational system with trade list selected

as you can see 25% is the risk of ruin

but since there are many overlapping trades i selected the portfolio equity changes and ran it again

and there is no risk of ruin

[JulianCohen]

What do they mean by overlapping trades? I couldn’t work that out

[Anonymous]

Said, that doesn’t really make sense to me, for a system like the MOC systems anyway where we are opening and closing a trade within the same day/bar.

I would assume the shuffling of trades, changing the order in which they occur, would only happen within the one and the same bar. It really wouldn’t make any sense that a trade made on a certain date, at a certain size, could be pulled back or used as a potential trade many years earlier in the sequence of trades, especially in a system of trades that are opened and closed within the same bar. What would be the relevance or usefulness of such a thing?

I would also assume that “overlapping” just means any trade that is open at the same time on any one single bar. This would mean that technically all our trades are overlapping as they are all opened and then closed on the one and the same bar. This again tells me that trades could well be shuffled around, deleted, duplicated a number of times etc, but all within the one and the same bar only. It would not, for example, take a trade performed on 12th June and then attempt to perform that trade on 11th June. I mean, why would it? That would completely defeat the purpose of the software and what we are trying to program/prove. That would not sound like a monte carlo to me. That would just sound like a complete bloody mess.

Said, I am not making my point for rotational systems or any other type of system that operates a trade beyond one single bar. I am only commenting from the MOC style system angle, where the trades are opened and closed within the one bar only. Also, I am definitely only referring to when using the CBT code. The CBT gives us the trade list, without the CBT code, Amibroker still creates a trade list, but it is randomised from all the possible trades within a given bar. For example on a 50 position MOC style system, perhaps today’s exploration has 200 possible orders. Without the ranking and CBT code, I would be confident the regular process randomly picks from the 200 possible trades and uses 50 of them for today’s trades. Therefore I think we must take an exception to how the Amibroker help file explains when it is best to use trade list mode or equity mode. My opinion is that perhaps the help file is written without the assumption that we are using some CBT code that helps give a definitive trade list for a single day of trades whereby the top ranked trades would always and forever used without exception in the same ranking/sequence/order. Of course I could be completely wrong as it is only an assumption.

[JulianCohen]

The one thing this discussion is showing me is that between a number of very smart people on this forum, we can’t really work out what the Amibroker Monte Carlo is really doing for sure. I agree with your premise Matthew, but we can’t say for certain that is what it is doing.

[Anonymous]

Absolutely is confusing. Good thing is we’ll surely work it out between us by brewing the ideas together here on the forum. I know I’d be stuffed if I had to be stuck on my own trying to work it out for myself.

[Author]

Posts