Select Page

# Out-of-sample Testing Using Monte Carlo Simulations

Apr 6, 2021

###### This post demonstrates the use of StrategyQuant’s Monte Carlo simulator to randomize historical prices and strategy parameters, helping you select robust strategies for live trading.

Robustness is a big deal in trading strategy development.

A robust strategy is insensitive to variations in price behaviour, meaning it will continue to perform well when market conditions change.

To quantify a strategy’s robustness, traders often include out-of-sample tests during development. This usually involves backtesting the strategy on a different market or timeframe.

But what if you want to run an out-of-sample test on the same market and timeframe?

# What is Monte Carlo Simulation?

Monte Carlo simulation uses repeated random sampling to produce different backtest outcomes.

It is most commonly used to randomize your backtest trade sequence, essentially producing a new equity curve with a different risk and return profile. This is a good approach to estimate your worst-case drawdowns.

The example below shows 25 equity curve simulations, each with a different trade sequence.

Apart from randomizing your trade sequence, StrategyQuant’s Monte Carlo simulator can also randomize your:

• Historical prices
• Strategy Parameters

Both of these methods effectively ‘shake’ your strategy to evaluate how much curve fitting has occurred over your development, much like how you shake a ladder before climbing it.

If your strategy’s performance suffers drastically due to changing prices or parameters, it has likely fallen victim to overfitting. Such strategies lack robustness and will likely not trade as well as their backtests indicate.

To demonstrate the use of these Monte Carlo tests, I’ll use the GBPJPY trend following strategy generated in the StrategyQuant development section.

This strategy trades on the hourly timeframe, but enters the market on a breakout of the previous week’s opening price. Trade management is minimal, with only a bar-based time stop and an ATR-based stop loss.

# Monte Carlo With Randomized Prices

This test creates variations in the historical prices used for backtesting.

## Setting Up

Let’s load the strategy and configure the Monte Carlo simulator.

Select the Monte Carlo Retest Methods option in StrategyQuant’s Retester module.

Under the Settings tab, let’s configure the following:

1. Number of simulations: 1000 is the maximum allowed. Unlike randomizing your trade sequence, a fresh backtest needs to be performed each time a new equity curve is created.
2. Use Full Sample: I’ll toggle this ON. By default, the simulations will only be performed on the in-sample portion of your data (configured under the Data tab). If you included an out-of-sample portion shown below, this portion will not be used for backtesting. Selecting Full Sample means all the available data will be used.
3. Backtest Precision: I recommend Selected Timeframe Only, which is equivalent to the ‘Open prices only’ model in MT4. The alternative is ‘1-minute data tick simulation’, which is equivalent to the default ‘Every tick’ model in MT4. This second model is much slower because it simulates tick movement by interpolating between the 1-minute OHLC prices.
4. Select the Randomize History Data option from the list of available simulations.
5. Probability Up/Max Up Change: Probability Up is the % likelihood that each bar in the price history will have one of its prices (open, high, low, close) increased. Max Up Change is the maximum allowable change, as a % of the 14-period ATR.  I’ll use 30% for both fields. So for example, if the ATR is 10 pips, there is a 30% likelihood that the price will increase by up to 3 pips. I recommend using similar values for downside price changes, since it is difficult to predict market bias.
6. Keep Connected: I’ll toggle this ON. This preserves any price gaps in the original data series.

Before proceeding, now’s a good time to introduce the concept of confidence levels.

Monte Carlo simulation is probability-based, meaning your results will differ slightly each time you run the simulation. It’s like flipping a fair coin 1000 times – you’re not going to get 500 heads every time. In theory, your backtest metric will converge towards a certain ‘true’ value after running an infinite number of simulations.

This is obviously an impractical approach. A good compromise would be to run a sample of simulations (1000 in this case), and use confidence levels to quantify the uncertainty arising from this smaller sample.

A confidence level refers to the probability that the sampled results contain a parameter’s true value. For this post, I’ll use the backtest’s return/maximum drawdown ratio (Ret/DD) as the parameter of interest.

Here’s how we apply confidence levels to the Ret/DD. The following table shows a typical result from a Monte Carlo simulation:

With a 95% confidence level, there is only a 5% probability that:

• The net profit will be lower than \$581
• The drawdown will be higher than \$157
• The Ret/DD will be lower than 3.91

Higher confidence levels result in greater deterioration of your Monte Carlo metrics, but these metrics are more likely to encompass your future performance.

With an understanding of confidence levels, it’s time to input the test filters. These will be useful if you’re using the Monte Carlo simulations to shortlist strategies. Head over to the Filtering tab.

I’ll only use one condition, which states that the Ret/DD at the 95% confidence level should be at least 50% of the original backtest’s Ret/DD. This left side value is setup as follows:

1. Metric: Select your metric of interest. I’ll use Return/Maximum Drawdown here. If you trade a small account and are concerned about capitalization, maximum drawdown is an alternative.
2. From Backtest: Select the relevant robustness test. Monte Carlo Retest Methods is what we’re using.
3. At Confidence Level: 95% is a commonly used level. You can use this to adjust the difficulty of your test. Even if you use a 100% confidence level, it doesn’t mean that future performance cannot be worse. Monte Carlo simulations are estimations, after all.

As for the right side value,

1. From Backtest: Similar to above, but this time we will be using the original backtest as a benchmark, so select Main Data.
2. Sample: Like the Use Full Sample option described above, we want to use all the available data, so select Full.
3. Direction: Select Both, since the strategy trades symmetrically on the long and short sides.
4. Result In: This depends on what units your metric uses. It is irrelevant here because Ret/DD doesn’t have units, although Money is selected by default.
5. Apply Percentage Ratio: Since we want the Monte Carlo Ret/DD to be at least 50% of the original, set the percentage to 50%

That’s it! Let’s run the simulation.

## Monte Carlo Simulation Results (Randomized Prices)

The characteristic ‘straw broom’ Monte Carlo chart is shown below.

There are 1000 additional equity curves, each created by backtesting on different historical data. As a consequence, the curves contain different numbers of trades and final equity values.

The original equity curve is mostly in the top third of the ‘broom.’ This means majority of the simulations resulted in worse outcomes, which is not great. Let’s pull up the confidence levels.

As we progress through the confidence levels, the Ret/DD tends to deteriorate significantly because we expect a simultaneous decrease in profit and increase in drawdown. The 5.03 Ret/DD at the 95% confidence level is 61% of the original value, so the test passes.

# Monte Carlo With Randomized Parameters

This test randomizes your strategy parameters. Robust strategies remain profitable over a large range of parameters.

This strategy only contains three parameters:

• Number of bars the entry buy/sell stops will be valid for
• Numbers of bars used for the time stop
• Multiple for the ATR-based stop loss

The fewer parameters you use, the more robust your strategies tend to be. Complex strategies fail in complex ways!

## Setting Up

The steps here are identical to those above, with the exception of the following:

1. Select Randomize Strategy Parameters from the list of available simulations.
2. Probability: This is the % likelihood that each parameter will be changed. I’ll input 30%.
3. Max Change: This is the maximum % change that each parameter will be subjected to. I’ll input 30%. For example, a 100-bar time stop can have its value changed to anything in the 70-130 range.
4. Symmetric Parameters: I’ll toggle this ON. This means the same parameters will be used for both the long and short sides.

## Monte Carlo Simulation Results (Randomized Parameters)

The ‘straw brooms’ look quite similar to those obtained from the randomized prices, with one key difference – there is a much larger variation in the number of trades per simulation. This is due to changes in the length of the time stop and size of the stop loss.

The 95% confidence level Ret/DD is slightly lower than that in the randomized prices simulation, but still passes our 50% filter.

So far the strategy seems robust; it has performed reasonably well in the face of changing prices and parameters. Being a perverse tester I can’t help but think: what if we randomize both prices and parameters simultaneously?

I did exactly that, and the chart looks more scattered, with even larger variation in the number of trades.

The 95% confidence level Ret/DD is down to 42% of the original, so if you want a more stringent robustness test, this is the way to go.

# Wrapping Up

Using Monte Carlo simulations to randomize your prices and/or strategy parameters is another way to do out-of-sample testing.

If your strategy’s performance does not deteriorate significantly as you progress through the confidence levels, it is likely you have a robust strategy at your fingertips.

Supplementing these simulations with other robustness tests, such as walk-forward optimization, is a great way to defeat overfitting.

If you want to check out StrategyQuant’s Monte Carlo capabilities, why not take advantage of its 14-day FREE trial?

Don’t forget to use coupon TACT to get USD 200 off StrategyQuant Pro!

## Popular Posts

### What is Considered Good Trading Performance?

Are you taking too much risk for too little return? Perhaps the professional hedge funds can help define good trading performance.

### Automated MACD Divergence Forex Trading Strategy

The MACD is a simple and effective momentum indicator. Here’s how you can save screen time by programming a MACD divergence strategy for the GBPJPY!

### Laguerre RSI Trend Following Strategy

The Laguerre RSI attempts to improve the responsiveness of the regular RSI, whilst keeping whipsaw trades to a minimum. Let’s see how well it detects short-term pullbacks for a trend following strategy!

### What Is the Kaufman Adaptive Moving Average?

The Kaufman Adaptive Moving Average is a unique indicator that automatically adapts to the market’s noise. Here I explain its inner workings and show you how to build a trend following strategy around it.

### What is Fixed Ratio Money Management?

Have you heard of fixed ratio money management? How does it compare to the popular fixed fractional approach? Here I’ll explain how fixed ratio works, and see how it stacks up against fixed fractional money management.

### Build a Diversified Portfolio With QuantAnalyzer

The ability to efficiently trade a diversified portfolio of strategies is one of the biggest advantages of algorithmic trading. Here we will use QuantAnalyzer’s Portfolio Master to build a portfolio consisting of high performing, uncorrelated strategies.

### What Is the QQE Indicator?

The QQE is a mysterious indicator that sometimes pops up in trading forums. Does it deserve a place alongside the more traditional momentum indicators like the RSI and CCI? Let’s add it to a trend following strategy to find out!

### Do Bollinger Bands + Candlestick Patterns Work?

Bollinger Bands are great at detecting overbought and oversold conditions. Let’s use them to develop a countertrend strategy, and then refine our entries using limit entries and candlestick patterns.

### How Good Are The Bollinger Bands’ Trailing Stops?

Trailing stop losses are a popular feature in many trend following systems. Bollinger Bands, the ever-popular technical indicator among retail traders, actually contain two inbuilt trailing stops. Are these any good? Let’s find out!

#### Make your money work for you!

Get promotions, trading ideas and strategy development tips delivered to your inbox!

## What’s the Best Time to Trade Forex?

The forex markets are open 24/5, but not all hours are created equal. Here I dissect my broker data to determine the best time to trade forex.

## Forex Weekend Gaps: Can You Exploit Them?

Have you noticed that forex weekend gaps usually reverse within 3 days? Here I’ll program a mean reversion strategy to exploit gaps over the last 18 years!

## Dynamic Position Sizing: Is It Time to Go Big?

Should you increase your lot sizes for higher probability trades? Let’s code a dynamic position sizing scheme to capture more outsized wins!

## Pivot Points: A Reliable Support & Resistance Indicator

Pivot points are the perfect tool if you trade using support & resistance. Here’s how to develop an automated pivot points forex strategy!

## Money Flow Index: An Improved RSI?

The Money Flow Index is sometimes called the volume-weighted RSI. Can it outperform the RSI in this trend following strategy?

## Automated Bollinger Bands Squeeze Forex Strategy

StrategyQuant’s BBWR indicator is the perfect tool to detect a Bollinger Bands squeeze. Here I explain how it’s calculated, and use it to program a breakout strategy for the AUDJPY!

## Automated Schaff Trend Cycle Forex Strategy

The Schaff Trend Cycle is a unique combination of the MACD and Stochastic indicators. Here’s how you can use it to improve your trend following results!

## Should You Use the Kelly Criterion for Forex Trading?

The Kelly criterion is a famous mathematical formula that attempts to maximize your long-term capital growth. In this post, I’ll apply it to a EURUSD breakout strategy and explain some of its potential shortcomings when applied to forex trading.

## Forex Intermarket Correlations: How Do You Exploit Them?

Knowledge of intermarket correlations can improve your forex trading win rate. Here I explain three important types of correlations, and how you can use them to benefit your trading.

A temporary trading pause can improve your win rate if you’re trend following a volatile market. Here I’ll program a trading pause into a simple breakout strategy, and test its effectiveness on the Widow Maker – the GBPJPY.

## Can Partial Profit Taking Benefit Trend Followers?

Partial profit taking is a dilemma often faced by long-term trend followers. Could this benefit your overall strategy performance?

Let’s test!

## Ride Breakouts With This Support & Resistance Indicator!

StrategyQuant offers a support & resistance ranking indicator to help detect upcoming breakouts. Use it to program an automated USDJPY strategy!

## Automated MACD Divergence Forex Trading Strategy

The MACD is a simple and effective momentum indicator. Here’s how you can save screen time by programming a MACD divergence strategy for the GBPJPY!

## Catch Breakouts With an Inside Bar Trend Strategy!

Inside bars can be a valuable predictor of trend continuation. Here’s how you can program an inside bar, and use it to improve your trend following strategy!

## Laguerre RSI Trend Following Strategy

The Laguerre RSI attempts to improve the responsiveness of the regular RSI, whilst keeping whipsaw trades to a minimum. Let’s see how well it detects short-term pullbacks for a trend following strategy!

## Bollinger Bands Free Bar Trading Strategy

Bollinger Bands are great at predicting upcoming price reversals. Here’s how you can use its ‘free bars’ to build a reliable Asian scalper!

## How to Use the Supertrend Indicator

Despite its cool name, the Supertrend indicator often seems to slip under the radar. Here I explain how it’s calculated, and combine it with moving averages to produce a simple trend following strategy.

## 5 Easy Trailing Stops to Boost Your Trend Following Results

Trailing stops are a popular feature in trend following strategies. They provide downside protection and allow you to ride the big trends. Try out these simple yet effective trailing stops for your next trend strategy!

## Book Review: Building Winning Algorithmic Trading Systems

Serious traders invest in quality education. Should you read “Building Winning Algorithmic Trading Systems” by Kevin Davey?

## Trade Slippage: How Can You Simulate and Minimize It?

Are you a victim of excessive trade slippage? Here’s how you can minimize slippage, and more realistically simulate it in your backtests!

## Edge Ratio: A Unique Way to Quantify Entry Profitability

Selecting a profitable entry is a critical step in strategy development. Here I’ll demonstrate how to use the Edge Ratio to maximize your profit potential.

## Using Maximum Adverse Excursion for Stop Loss Placement

A catastrophic stop loss is a vital risk management tool for many traders. Here I’ll show you how to optimize your stop loss distance using maximum adverse excursion.

## Multiple Timeframe Backtesting – A Quick Robustness Test

Multiple timeframe backtesting can be a valuable addition to your strategy development workflow. Here I explain why you should do it, and how to conveniently do it in MT4 and StrategyQuant.

## How to Get a Realistic Backtest Spread

Your choice of backtest spread can certainly make or break a strategy. This post will show you how to study the intraday spread variations of your market, and suggest several ways to avoid paying ridiculous spreads.

Risk of ruin is a useful metric to help develop trading strategies that suit your risk appetite. This post explains how to calculate your risk of ruin, and how to use it to improve your trading!

## Do You Know Your Strategy’s Optimization Profile?

Your strategy’s optimization profile often reveals its robustness, helping you select strategies that will remain profitable in live trading. Here I explain why an optimization profile is important, and how you can easily obtain one using StrategyQuant’s optimizer.

## Which MT4 Backtest Report Metrics Should You Use?

Understanding your backtest report is an essential part of being a successful strategy developer. Here I explain what the numbers mean, and how you can make use of each metric during strategy development.

We all want a large sample of trades in our backtests, but practical limitations such as data availability often get in the way. Here I’ll explain why 30 trades is insufficient, and how you can use standard error to quantify the uncertainty arising from a small sample size.

## Which Forex Markets Are the Best for Trend Following?

They say the trend is your friend, but how do you find your friends in the forex jungle? This post shows you how to use the ADX indicator to identify forex markets and timeframes that trend well.

## Build a Diversified Portfolio With QuantAnalyzer

The ability to efficiently trade a diversified portfolio of strategies is one of the biggest advantages of algorithmic trading. Here we will use QuantAnalyzer’s Portfolio Master to build a portfolio consisting of high performing, uncorrelated strategies.

## Strategy Optimization Using MT4

How do you improve your trading strategy in MT4? This post will show you how to optimize the entry and exit parameters for a moving average crossover strategy. Finally, an intraday time filter will be added to help avoid false breakouts.

## Debugging & Backtesting Using MT4

With a fresh algorithm at your fingertips, how do you verify that it has been programmed correctly? This guide will show you how to use Metatrader 4’s visual backtester to debug and backtest your strategy.

Converting your trading idea into an algorithm is the first step towards reaping the benefits of automated trading. This guide will cover the creation of a simple moving average crossover algorithm, without any actual programming.

## What Is Drawdown in Trading?

Are you getting a comprehensive assessment of your strategy’s downside? This post will discuss several methods to measure drawdowns, helping you build and select strategies that better suit your risk appetite.

## How to Select the Best Trading Strategy Entry

With an abundance of technical indicators available, selecting your strategy’s entry conditions can be overwhelming. This post will illustrate a method to graphically compare the profitability of different entries.

## Who Are the Major Players in the Forex Market?

With its daily turnover of over \$6 trillion, who are the major players in the forex market? Let’s see how retail traders stack up!

## US Nonfarm Payrolls: What’s Considered a Big Miss?

A big miss in the US nonfarm payrolls can wreak havoc in the markets. What’s considered a big miss? I analyze historical values to find out!

## What’s the Best Time to Trade Forex?

The forex markets are open 24/5, but not all hours are created equal. Here I dissect my broker data to determine the best time to trade forex.

## What is Considered Good Trading Performance?

Are you taking too much risk for too little return? Perhaps the professional hedge funds can help define good trading performance.

## 5 Forex Day Trading Challenges & How to Overcome Them

Forex day trading seems to have a particular appeal to new traders. Here I highlight five hidden challenges of day trading, and offer some suggestions on how to overcome them.

## How to Find a Real Trading Guru

Every day I come across a trading guru offering educational content on the internet. Many of them speak of huge returns with minimal effort. Should these be trusted? Here’s some tips on how to separate the wheat from the chaff.

## How to Enjoy Stress-Free Trading

Trading is a great way to make some additional income, but not if you’re constantly pulling your hair out. Here I offer 7 tips to help make your trading profitable and stress-free.

## How to Select the Best Forex VPS

A virtual private server (VPS) is a virtual computer that you can rent and access remotely. It provides a reliable platform on which to execute your forex strategies. This post will help you decide whether you need a VPS, and show you how to select an optimal VPS.