Out-of-sample Testing Using Monte Carlo Simulations

Apr 6, 2021

Traders often use Monte Carlo simulations to estimate worst-case drawdowns, but did you know they can be used for out-of-sample testing too?
This post demonstrates the use of StrategyQuant’s Monte Carlo simulator to randomize historical prices and strategy parameters, helping you select robust strategies for live trading.

Robustness is a big deal in trading strategy development.

A robust strategy is insensitive to variations in price behaviour, meaning it will continue to perform well when market conditions change.

To quantify a strategy’s robustness, traders often include out-of-sample tests during development. This usually involves backtesting the strategy on a different market or timeframe.

But what if you want to run an out-of-sample test on the same market and timeframe?

StrategyQuant’s Monte Carlo simulator can help you with this.

What is Monte Carlo Simulation?

Monte Carlo simulation uses repeated random sampling to produce different backtest outcomes.

It is most commonly used to randomize your backtest trade sequence, essentially producing a new equity curve with a different risk and return profile. This is a good approach to estimate your worst-case drawdowns.

The example below shows 25 equity curve simulations, each with a different trade sequence.

Apart from randomizing your trade sequence, StrategyQuant’s Monte Carlo simulator can also randomize your:

  • Historical prices
  • Strategy Parameters

Both of these methods effectively ‘shake’ your strategy to evaluate how much curve fitting has occurred over your development, much like how you shake a ladder before climbing it.

If your strategy’s performance suffers drastically due to changing prices or parameters, it has likely fallen victim to overfitting. Such strategies lack robustness and will likely not trade as well as their backtests indicate.

To demonstrate the use of these Monte Carlo tests, I’ll use the GBPJPY trend following strategy generated in the StrategyQuant development section.

This strategy trades on the hourly timeframe, but enters the market on a breakout of the previous week’s opening price. Trade management is minimal, with only a bar-based time stop and an ATR-based stop loss. 

You can download this strategy here

Monte Carlo With Randomized Prices

This test creates variations in the historical prices used for backtesting.

Setting Up

Let’s load the strategy and configure the Monte Carlo simulator.

Select the Monte Carlo Retest Methods option in StrategyQuant’s Retester module. 

Under the Settings tab, let’s configure the following:

  1. Number of simulations: 1000 is the maximum allowed. Unlike randomizing your trade sequence, a fresh backtest needs to be performed each time a new equity curve is created.
  2. Use Full Sample: I’ll toggle this ON. By default, the simulations will only be performed on the in-sample portion of your data (configured under the Data tab). If you included an out-of-sample portion shown below, this portion will not be used for backtesting. Selecting Full Sample means all the available data will be used.
  3. Backtest Precision: I recommend Selected Timeframe Only, which is equivalent to the ‘Open prices only’ model in MT4. The alternative is ‘1-minute data tick simulation’, which is equivalent to the default ‘Every tick’ model in MT4. This second model is much slower because it simulates tick movement by interpolating between the 1-minute OHLC prices.
  4. Select the Randomize History Data option from the list of available simulations.
  5. Probability Up/Max Up Change: Probability Up is the % likelihood that each bar in the price history will have one of its prices (open, high, low, close) increased. Max Up Change is the maximum allowable change, as a % of the 14-period ATR.  I’ll use 30% for both fields. So for example, if the ATR is 10 pips, there is a 30% likelihood that the price will increase by up to 3 pips. I recommend using similar values for downside price changes, since it is difficult to predict market bias. 
  6. Keep Connected: I’ll toggle this ON. This preserves any price gaps in the original data series.

Before proceeding, now’s a good time to introduce the concept of confidence levels.

Monte Carlo simulation is probability-based, meaning your results will differ slightly each time you run the simulation. It’s like flipping a fair coin 1000 times – you’re not going to get 500 heads every time. In theory, your backtest metric will converge towards a certain ‘true’ value after running an infinite number of simulations.

This is obviously an impractical approach. A good compromise would be to run a sample of simulations (1000 in this case), and use confidence levels to quantify the uncertainty arising from this smaller sample.

A confidence level refers to the probability that the sampled results contain a parameter’s true value. For this post, I’ll use the backtest’s return/maximum drawdown ratio (Ret/DD) as the parameter of interest. 

Here’s how we apply confidence levels to the Ret/DD. The following table shows a typical result from a Monte Carlo simulation:

With a 95% confidence level, there is only a 5% probability that:

  • The net profit will be lower than $581
  • The drawdown will be higher than $157
  • The Ret/DD will be lower than 3.91

Higher confidence levels result in greater deterioration of your Monte Carlo metrics, but these metrics are more likely to encompass your future performance.

With an understanding of confidence levels, it’s time to input the test filters. These will be useful if you’re using the Monte Carlo simulations to shortlist strategies. Head over to the Filtering tab.

I’ll only use one condition, which states that the Ret/DD at the 95% confidence level should be at least 50% of the original backtest’s Ret/DD. This left side value is setup as follows:

  1. Metric: Select your metric of interest. I’ll use Return/Maximum Drawdown here. If you trade a small account and are concerned about capitalization, maximum drawdown is an alternative.
  2. From Backtest: Select the relevant robustness test. Monte Carlo Retest Methods is what we’re using.
  3. At Confidence Level: 95% is a commonly used level. You can use this to adjust the difficulty of your test. Even if you use a 100% confidence level, it doesn’t mean that future performance cannot be worse. Monte Carlo simulations are estimations, after all. 

As for the right side value,

  1. From Backtest: Similar to above, but this time we will be using the original backtest as a benchmark, so select Main Data.
  2. Sample: Like the Use Full Sample option described above, we want to use all the available data, so select Full.
  3. Direction: Select Both, since the strategy trades symmetrically on the long and short sides. 
  4. Result In: This depends on what units your metric uses. It is irrelevant here because Ret/DD doesn’t have units, although Money is selected by default. 
  5. Apply Percentage Ratio: Since we want the Monte Carlo Ret/DD to be at least 50% of the original, set the percentage to 50%

That’s it! Let’s run the simulation.

Monte Carlo Simulation Results (Randomized Prices)

The characteristic ‘straw broom’ Monte Carlo chart is shown below.

There are 1000 additional equity curves, each created by backtesting on different historical data. As a consequence, the curves contain different numbers of trades and final equity values. 

The original equity curve is mostly in the top third of the ‘broom.’ This means majority of the simulations resulted in worse outcomes, which is not great. Let’s pull up the confidence levels.

As we progress through the confidence levels, the Ret/DD tends to deteriorate significantly because we expect a simultaneous decrease in profit and increase in drawdown. The 5.03 Ret/DD at the 95% confidence level is 61% of the original value, so the test passes.

Monte Carlo With Randomized Parameters

This test randomizes your strategy parameters. Robust strategies remain profitable over a large range of parameters. 

This strategy only contains three parameters:

  • Number of bars the entry buy/sell stops will be valid for
  • Numbers of bars used for the time stop
  • Multiple for the ATR-based stop loss

The fewer parameters you use, the more robust your strategies tend to be. Complex strategies fail in complex ways!

Setting Up

The steps here are identical to those above, with the exception of the following:

  1. Select Randomize Strategy Parameters from the list of available simulations.
  2. Probability: This is the % likelihood that each parameter will be changed. I’ll input 30%.
  3. Max Change: This is the maximum % change that each parameter will be subjected to. I’ll input 30%. For example, a 100-bar time stop can have its value changed to anything in the 70-130 range. 
  4. Symmetric Parameters: I’ll toggle this ON. This means the same parameters will be used for both the long and short sides. 

Monte Carlo Simulation Results (Randomized Parameters)

The ‘straw brooms’ look quite similar to those obtained from the randomized prices, with one key difference – there is a much larger variation in the number of trades per simulation. This is due to changes in the length of the time stop and size of the stop loss.

The 95% confidence level Ret/DD is slightly lower than that in the randomized prices simulation, but still passes our 50% filter. 

So far the strategy seems robust; it has performed reasonably well in the face of changing prices and parameters. Being a perverse tester I can’t help but think: what if we randomize both prices and parameters simultaneously?

I did exactly that, and the chart looks more scattered, with even larger variation in the number of trades. 

The 95% confidence level Ret/DD is down to 42% of the original, so if you want a more stringent robustness test, this is the way to go.

Wrapping Up

Using Monte Carlo simulations to randomize your prices and/or strategy parameters is another way to do out-of-sample testing. 

If your strategy’s performance does not deteriorate significantly as you progress through the confidence levels, it is likely you have a robust strategy at your fingertips. 

Supplementing these simulations with other robustness tests, such as walk-forward optimization, is a great way to defeat overfitting.

If you want to check out StrategyQuant’s Monte Carlo capabilities, why not take advantage of its 14-day FREE trial?

Don’t forget to use coupon TACT to get USD 200 off StrategyQuant Pro!

Powered By

Development Platform

Forex VPS

FXVM Forex VPS

Popular Posts

Laguerre RSI Trend Following Strategy

The Laguerre RSI attempts to improve the responsiveness of the regular RSI, whilst keeping whipsaw trades to a minimum. Let’s see how well it detects short-term pullbacks for a trend following strategy!

read more

What is Fixed Ratio Money Management?

Have you heard of fixed ratio money management? How does it compare to the popular fixed fractional approach? Here I’ll explain how fixed ratio works, and see how it stacks up against fixed fractional money management.

read more

Build a Diversified Portfolio With QuantAnalyzer

The ability to efficiently trade a diversified portfolio of strategies is one of the biggest advantages of algorithmic trading. Here we will use QuantAnalyzer’s Portfolio Master to build a portfolio consisting of high performing, uncorrelated strategies.

read more

What Is the QQE Indicator?

The QQE is a mysterious indicator that sometimes pops up in trading forums. Does it deserve a place alongside the more traditional momentum indicators like the RSI and CCI? Let’s add it to a trend following strategy to find out!

read more

Make your money work for you!

Get promotions, trading ideas and strategy development tips delivered to your inbox!

Comments

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Trading Strategies

What’s the Best Time to Trade Forex?

What’s the Best Time to Trade Forex?

The forex markets are open 24/5, but not all hours are created equal. Here I dissect my broker data to determine the best time to trade forex.

Forex Weekend Gaps: Can You Exploit Them?

Forex Weekend Gaps: Can You Exploit Them?

Have you noticed that forex weekend gaps usually reverse within 3 days? Here I’ll program a mean reversion strategy to exploit gaps over the last 18 years!

Money Flow Index: An Improved RSI?

Money Flow Index: An Improved RSI?

The Money Flow Index is sometimes called the volume-weighted RSI. Can it outperform the RSI in this trend following strategy?

Automated Bollinger Bands Squeeze Forex Strategy

Automated Bollinger Bands Squeeze Forex Strategy

StrategyQuant’s BBWR indicator is the perfect tool to detect a Bollinger Bands squeeze. Here I explain how it’s calculated, and use it to program a breakout strategy for the AUDJPY!

Should You Use the Kelly Criterion for Forex Trading?

Should You Use the Kelly Criterion for Forex Trading?

The Kelly criterion is a famous mathematical formula that attempts to maximize your long-term capital growth. In this post, I’ll apply it to a EURUSD breakout strategy and explain some of its potential shortcomings when applied to forex trading.

Can a Trading Pause Improve Your Trend Following Results?

Can a Trading Pause Improve Your Trend Following Results?

A temporary trading pause can improve your win rate if you’re trend following a volatile market. Here I’ll program a trading pause into a simple breakout strategy, and test its effectiveness on the Widow Maker – the GBPJPY.

Laguerre RSI Trend Following Strategy

Laguerre RSI Trend Following Strategy

The Laguerre RSI attempts to improve the responsiveness of the regular RSI, whilst keeping whipsaw trades to a minimum. Let’s see how well it detects short-term pullbacks for a trend following strategy!

How to Use the Supertrend Indicator

How to Use the Supertrend Indicator

Despite its cool name, the Supertrend indicator often seems to slip under the radar. Here I explain how it’s calculated, and combine it with moving averages to produce a simple trend following strategy.

Strategy Development

Do You Know Your System Quality Number?

Do You Know Your System Quality Number?

The System Quality Number measures the profitability & consistency of your trading system. Here’s how to calculate your SQN and use it to improve your trading!

How to Get a Realistic Backtest Spread

How to Get a Realistic Backtest Spread

Your choice of backtest spread can certainly make or break a strategy. This post will show you how to study the intraday spread variations of your market, and suggest several ways to avoid paying ridiculous spreads.

Do You Know Your Strategy’s Optimization Profile?

Do You Know Your Strategy’s Optimization Profile?

Your strategy’s optimization profile often reveals its robustness, helping you select strategies that will remain profitable in live trading. Here I explain why an optimization profile is important, and how you can easily obtain one using StrategyQuant’s optimizer.

Which MT4 Backtest Report Metrics Should You Use?

Which MT4 Backtest Report Metrics Should You Use?

Understanding your backtest report is an essential part of being a successful strategy developer. Here I explain what the numbers mean, and how you can make use of each metric during strategy development.

How Many Trades Should Your Backtest Have?

How Many Trades Should Your Backtest Have?

We all want a large sample of trades in our backtests, but practical limitations such as data availability often get in the way. Here I’ll explain why 30 trades is insufficient, and how you can use standard error to quantify the uncertainty arising from a small sample size.

Build a Diversified Portfolio With QuantAnalyzer

Build a Diversified Portfolio With QuantAnalyzer

The ability to efficiently trade a diversified portfolio of strategies is one of the biggest advantages of algorithmic trading. Here we will use QuantAnalyzer’s Portfolio Master to build a portfolio consisting of high performing, uncorrelated strategies.

Strategy Optimization Using MT4

Strategy Optimization Using MT4

How do you improve your trading strategy in MT4? This post will show you how to optimize the entry and exit parameters for a moving average crossover strategy. Finally, an intraday time filter will be added to help avoid false breakouts.

Debugging & Backtesting Using MT4

Debugging & Backtesting Using MT4

With a fresh algorithm at your fingertips, how do you verify that it has been programmed correctly? This guide will show you how to use Metatrader 4’s visual backtester to debug and backtest your strategy.

Create Your Trading Algorithm in 15 Minutes (FREE)

Create Your Trading Algorithm in 15 Minutes (FREE)

Converting your trading idea into an algorithm is the first step towards reaping the benefits of automated trading. This guide will cover the creation of a simple moving average crossover algorithm, without any actual programming.

What Is Drawdown in Trading?

What Is Drawdown in Trading?

Are you getting a comprehensive assessment of your strategy’s downside? This post will discuss several methods to measure drawdowns, helping you build and select strategies that better suit your risk appetite.

How to Select the Best Trading Strategy Entry

How to Select the Best Trading Strategy Entry

With an abundance of technical indicators available, selecting your strategy’s entry conditions can be overwhelming. This post will illustrate a method to graphically compare the profitability of different entries.

Live Trading

What’s the Best Time to Trade Forex?

What’s the Best Time to Trade Forex?

The forex markets are open 24/5, but not all hours are created equal. Here I dissect my broker data to determine the best time to trade forex.

How to Find a Real Trading Guru

How to Find a Real Trading Guru

Every day I come across a trading guru offering educational content on the internet. Many of them speak of huge returns with minimal effort. Should these be trusted? Here’s some tips on how to separate the wheat from the chaff.

How to Enjoy Stress-Free Trading

How to Enjoy Stress-Free Trading

Trading is a great way to make some additional income, but not if you’re constantly pulling your hair out. Here I offer 7 tips to help make your trading profitable and stress-free.

How to Select the Best Forex VPS

How to Select the Best Forex VPS

A virtual private server (VPS) is a virtual computer that you can rent and access remotely. It provides a reliable platform on which to execute your forex strategies. This post will help you decide whether you need a VPS, and show you how to select an optimal VPS.

Make your money work for you!

Make your money work for you!

 

Get trading ideas and strategy development tips delivered to your inbox!

Thanks for subscribing!

Pin It on Pinterest

Share This