Select Page

# Out-of-sample Testing Using Monte Carlo Simulations

Apr 6, 2021

###### This post demonstrates the use of StrategyQuant’s Monte Carlo simulator to randomize historical prices and strategy parameters, helping you select robust strategies for live trading.

Robustness is a big deal in trading strategy development.

A robust strategy is insensitive to variations in price behaviour, meaning it will continue to perform well when market conditions change.

To quantify a strategy’s robustness, traders often include out-of-sample tests during development. This usually involves backtesting the strategy on a different market or timeframe.

But what if you want to run an out-of-sample test on the same market and timeframe?

# What is Monte Carlo Simulation?

Monte Carlo simulation uses repeated random sampling to produce different backtest outcomes.

It is most commonly used to randomize your backtest trade sequence, essentially producing a new equity curve with a different risk and return profile. This is a good approach to estimate your worst-case drawdowns.

The example below shows 25 equity curve simulations, each with a different trade sequence.

Apart from randomizing your trade sequence, StrategyQuant’s Monte Carlo simulator can also randomize your:

• Historical prices
• Strategy Parameters

Both of these methods effectively ‘shake’ your strategy to evaluate how much curve fitting has occurred over your development, much like how you shake a ladder before climbing it.

If your strategy’s performance suffers drastically due to changing prices or parameters, it has likely fallen victim to overfitting. Such strategies lack robustness and will likely not trade as well as their backtests indicate.

To demonstrate the use of these Monte Carlo tests, I’ll use the GBPJPY trend following strategy generated in the StrategyQuant development section.

This strategy trades on the hourly timeframe, but enters the market on a breakout of the previous week’s opening price. Trade management is minimal, with only a bar-based time stop and an ATR-based stop loss.

# Monte Carlo With Randomized Prices

This test creates variations in the historical prices used for backtesting.

## Setting Up

Let’s load the strategy and configure the Monte Carlo simulator.

Select the Monte Carlo Retest Methods option in StrategyQuant’s Retester module.

Under the Settings tab, let’s configure the following:

1. Number of simulations: 1000 is the maximum allowed. Unlike randomizing your trade sequence, a fresh backtest needs to be performed each time a new equity curve is created.
2. Use Full Sample: I’ll toggle this ON. By default, the simulations will only be performed on the in-sample portion of your data (configured under the Data tab). If you included an out-of-sample portion shown below, this portion will not be used for backtesting. Selecting Full Sample means all the available data will be used.
3. Backtest Precision: I recommend Selected Timeframe Only, which is equivalent to the ‘Open prices only’ model in MT4. The alternative is ‘1-minute data tick simulation’, which is equivalent to the default ‘Every tick’ model in MT4. This second model is much slower because it simulates tick movement by interpolating between the 1-minute OHLC prices.
4. Select the Randomize History Data option from the list of available simulations.
5. Probability Up/Max Up Change: Probability Up is the % likelihood that each bar in the price history will have one of its prices (open, high, low, close) increased. Max Up Change is the maximum allowable change, as a % of the 14-period ATR.  I’ll use 30% for both fields. So for example, if the ATR is 10 pips, there is a 30% likelihood that the price will increase by up to 3 pips. I recommend using similar values for downside price changes, since it is difficult to predict market bias.
6. Keep Connected: I’ll toggle this ON. This preserves any price gaps in the original data series.

Before proceeding, now’s a good time to introduce the concept of confidence levels.

Monte Carlo simulation is probability-based, meaning your results will differ slightly each time you run the simulation. It’s like flipping a fair coin 1000 times – you’re not going to get 500 heads every time. In theory, your backtest metric will converge towards a certain ‘true’ value after running an infinite number of simulations.

This is obviously an impractical approach. A good compromise would be to run a sample of simulations (1000 in this case), and use confidence levels to quantify the uncertainty arising from this smaller sample.

A confidence level refers to the probability that the sampled results contain a parameter’s true value. For this post, I’ll use the backtest’s return/maximum drawdown ratio (Ret/DD) as the parameter of interest.

Here’s how we apply confidence levels to the Ret/DD. The following table shows a typical result from a Monte Carlo simulation:

With a 95% confidence level, there is only a 5% probability that:

• The net profit will be lower than \$581
• The drawdown will be higher than \$157
• The Ret/DD will be lower than 3.91

Higher confidence levels result in greater deterioration of your Monte Carlo metrics, but these metrics are more likely to encompass your future performance.

With an understanding of confidence levels, it’s time to input the test filters. These will be useful if you’re using the Monte Carlo simulations to shortlist strategies. Head over to the Filtering tab.

I’ll only use one condition, which states that the Ret/DD at the 95% confidence level should be at least 50% of the original backtest’s Ret/DD. This left side value is setup as follows:

1. Metric: Select your metric of interest. I’ll use Return/Maximum Drawdown here. If you trade a small account and are concerned about capitalization, maximum drawdown is an alternative.
2. From Backtest: Select the relevant robustness test. Monte Carlo Retest Methods is what we’re using.
3. At Confidence Level: 95% is a commonly used level. You can use this to adjust the difficulty of your test. Even if you use a 100% confidence level, it doesn’t mean that future performance cannot be worse. Monte Carlo simulations are estimations, after all.

As for the right side value,

1. From Backtest: Similar to above, but this time we will be using the original backtest as a benchmark, so select Main Data.
2. Sample: Like the Use Full Sample option described above, we want to use all the available data, so select Full.
3. Direction: Select Both, since the strategy trades symmetrically on the long and short sides.
4. Result In: This depends on what units your metric uses. It is irrelevant here because Ret/DD doesn’t have units, although Money is selected by default.
5. Apply Percentage Ratio: Since we want the Monte Carlo Ret/DD to be at least 50% of the original, set the percentage to 50%

That’s it! Let’s run the simulation.

## Monte Carlo Simulation Results (Randomized Prices)

The characteristic ‘straw broom’ Monte Carlo chart is shown below.

There are 1000 additional equity curves, each created by backtesting on different historical data. As a consequence, the curves contain different numbers of trades and final equity values.

The original equity curve is mostly in the top third of the ‘broom.’ This means majority of the simulations resulted in worse outcomes, which is not great. Let’s pull up the confidence levels.

As we progress through the confidence levels, the Ret/DD tends to deteriorate significantly because we expect a simultaneous decrease in profit and increase in drawdown. The 5.03 Ret/DD at the 95% confidence level is 61% of the original value, so the test passes.

# Monte Carlo With Randomized Parameters

This test randomizes your strategy parameters. Robust strategies remain profitable over a large range of parameters.

This strategy only contains three parameters:

• Number of bars the entry buy/sell stops will be valid for
• Numbers of bars used for the time stop
• Multiple for the ATR-based stop loss

The fewer parameters you use, the more robust your strategies tend to be. Complex strategies fail in complex ways!

## Setting Up

The steps here are identical to those above, with the exception of the following:

1. Select Randomize Strategy Parameters from the list of available simulations.
2. Probability: This is the % likelihood that each parameter will be changed. I’ll input 30%.
3. Max Change: This is the maximum % change that each parameter will be subjected to. I’ll input 30%. For example, a 100-bar time stop can have its value changed to anything in the 70-130 range.
4. Symmetric Parameters: I’ll toggle this ON. This means the same parameters will be used for both the long and short sides.

## Monte Carlo Simulation Results (Randomized Parameters)

The ‘straw brooms’ look quite similar to those obtained from the randomized prices, with one key difference – there is a much larger variation in the number of trades per simulation. This is due to changes in the length of the time stop and size of the stop loss.

The 95% confidence level Ret/DD is slightly lower than that in the randomized prices simulation, but still passes our 50% filter.

So far the strategy seems robust; it has performed reasonably well in the face of changing prices and parameters. Being a perverse tester I can’t help but think: what if we randomize both prices and parameters simultaneously?

I did exactly that, and the chart looks more scattered, with even larger variation in the number of trades.

The 95% confidence level Ret/DD is down to 42% of the original, so if you want a more stringent robustness test, this is the way to go.

# Wrapping Up

Using Monte Carlo simulations to randomize your prices and/or strategy parameters is another way to do out-of-sample testing.

If your strategy’s performance does not deteriorate significantly as you progress through the confidence levels, it is likely you have a robust strategy at your fingertips.

Supplementing these simulations with other robustness tests, such as walk-forward optimization, is a great way to defeat overfitting.

If you want to check out StrategyQuant’s Monte Carlo capabilities, why not take advantage of its 14-day FREE trial?

Don’t forget to use coupon TACT to get USD 200 off StrategyQuant Pro!

Want to develop a portfolio of automated trading strategies?

Supercharge your strategy development with StrategyQuant

Access 14-day FREE trial here!

Get up to USD 300 discount!

Strategies need improvement?

Use QuantAnalyzer’s powerful analysis tools

Try the FREE version here!

Get 20% Discount here!

## Automated MACD Divergence Forex Trading Strategy

The MACD is a simple and effective momentum indicator. Here’s how you can save screen time by programming a MACD divergence strategy for the GBPJPY!

## Laguerre RSI Trend Following Strategy

The Laguerre RSI attempts to improve the responsiveness of the regular RSI, whilst keeping whipsaw trades to a minimum. Let’s see how well it detects short-term pullbacks for a trend following strategy!

## What Is the Kaufman Adaptive Moving Average?

The Kaufman Adaptive Moving Average is a unique indicator that automatically adapts to the market’s noise. Here I explain its inner workings and show you how to build a trend following strategy around it.

## What is Fixed Ratio Money Management?

Have you heard of fixed ratio money management? How does it compare to the popular fixed fractional approach? Here I’ll explain how fixed ratio works, and see how it stacks up against fixed fractional money management.

## Build a Diversified Portfolio With QuantAnalyzer

The ability to efficiently trade a diversified portfolio of strategies is one of the biggest advantages of algorithmic trading. Here we will use QuantAnalyzer’s Portfolio Master to build a portfolio consisting of high performing, uncorrelated strategies.

## What Is the QQE Indicator?

The QQE is a mysterious indicator that sometimes pops up in trading forums. Does it deserve a place alongside the more traditional momentum indicators like the RSI and CCI? Let’s add it to a trend following strategy to find out!

## Do Bollinger Bands + Candlestick Patterns Work?

Bollinger Bands are great at detecting overbought and oversold conditions. Let’s use them to develop a countertrend strategy, and then refine our entries using limit entries and candlestick patterns.

## How Good Are The Bollinger Bands’ Trailing Stops?

Trailing stop losses are a popular feature in many trend following systems. Bollinger Bands, the ever-popular technical indicator among retail traders, actually contain two inbuilt trailing stops. Are these any good? Let’s find out!

Hey there, Wayne here! I’m on a mission to develop robust algorithmic trading strategies for the forex markets. Trading Tact is where I share my trading methods and insights.

###### Have a Question?

Want to develop a portfolio of automated trading strategies?

Supercharge your strategy development with StrategyQuant

Access 14-day FREE trial here!

Get up to USD 300 discount!

Strategies need improvement?

Use QuantAnalyzer’s powerful analysis tools

Try the FREE version here!

Get 20% Discount here!

## Forex Weekend Gaps: Can You Exploit Them?

Have you noticed that forex weekend gaps usually reverse within 3 days? Here I’ll program a mean reversion strategy to exploit gaps over the last 18 years!

## Dynamic Position Sizing: Is It Time to Go Big?

Should you increase your lot sizes for higher probability trades? Let’s code a dynamic position sizing scheme to capture more outsized wins!

## Pivot Points: A Reliable Support & Resistance Indicator

Pivot points are the perfect tool if you trade using support & resistance. Here’s how to develop an automated pivot points forex strategy!

## Money Flow Index: An Improved RSI?

The Money Flow Index is sometimes called the volume-weighted RSI. Can it outperform the RSI in this trend following strategy?

## Trade Slippage: How Can You Simulate and Minimize It?

Are you a victim of excessive trade slippage? Here’s how you can minimize slippage, and more realistically simulate it in your backtests!

## Edge Ratio: A Unique Way to Quantify Entry Profitability

Selecting a profitable entry is a critical step in strategy development. Here I’ll demonstrate how to use the Edge Ratio to maximize your profit potential.

## Automated Bollinger Bands Squeeze Forex Strategy

StrategyQuant’s BBWR indicator is the perfect tool to detect a Bollinger Bands squeeze. Here I explain how it’s calculated, and use it to program a breakout strategy for the AUDJPY!

## Automated Schaff Trend Cycle Forex Strategy

The Schaff Trend Cycle is a unique combination of the MACD and Stochastic indicators. Here’s how you can use it to improve your trend following results!

## Should You Use the Kelly Criterion for Forex Trading?

The Kelly criterion is a famous mathematical formula that attempts to maximize your long-term capital growth. In this post, I’ll apply it to a EURUSD breakout strategy and explain some of its potential shortcomings when applied to forex trading.

## Forex Intermarket Correlations: How Do You Exploit Them?

Knowledge of intermarket correlations can improve your forex trading win rate. Here I explain three important types of correlations, and how you can use them to benefit your trading.

## 5 Forex Day Trading Challenges & How to Overcome Them

Forex day trading seems to have a particular appeal to new traders. Here I highlight five hidden challenges of day trading, and offer some suggestions on how to overcome them.

A temporary trading pause can improve your win rate if you’re trend following a volatile market. Here I’ll program a trading pause into a simple breakout strategy, and test its effectiveness on the Widow Maker – the GBPJPY.

## Can Partial Profit Taking Benefit Trend Followers?

Partial profit taking is a dilemma often faced by long-term trend followers. Could this benefit your overall strategy performance?

Let’s test!

## Using Maximum Adverse Excursion for Stop Loss Placement

A catastrophic stop loss is a vital risk management tool for many traders. Here I’ll show you how to optimize your stop loss distance using maximum adverse excursion.

Hey there, Wayne here! I’m on a mission to develop robust algorithmic trading strategies for the forex markets. Trading Tact is where I share my trading methods and insights.