Machine learning and simulation - Monte carlo simulation and...

Learning Outcomes

This article explains Monte Carlo simulation and scenario analysis in machine learning–based portfolio and risk management for the CFA Level 2 exam, including:

• Differentiating Monte Carlo simulation, historical simulation, and scenario analysis, and selecting an appropriate approach for a given portfolio risk measurement task.
• Describing how input distributions, parameter choices, and dependence structures generate simulated paths and influence forecasts of returns, volatility, and tail losses.
• Designing and evaluating Monte Carlo simulations, covering variable selection, calibration windows, dependence modeling, number of runs, and diagnostic checks for model robustness.
• Estimating and interpreting VaR, CVaR, maximum drawdown, and related tail‑risk statistics from simulated output distributions in typical CFA Level 2 question formats.
• Contrasting sensitivity analysis with scenario analysis and explaining their roles in stress testing machine learning and factor‑based strategies.
• Identifying weaknesses and exam‑relevant traps in backtests and simulations, such as overreliance on historical data, mis‑specified distributions, and failure to capture regime shifts or extreme but plausible scenarios.

CFA Level 2 Syllabus

For the CFA Level 2 exam, you are required to understand simulation-based risk analysis and its role in machine learning for investment decisions, with a focus on the following syllabus points:

• Comparing parametric, historical simulation, and Monte Carlo approaches to estimating VaR and other risk measures.
• Describing inputs and decisions in simulation, and interpreting the output of simulations used to assess portfolio risk.
• Describing and contrasting sensitivity risk measures and scenario risk measures, and explaining their use in risk management and stress testing.
• Evaluating and interpreting historical scenario analysis and simulation results in the context of backtesting investment strategies.

Test Your Knowledge

Attempt these questions before reading this article. If you find some difficult or cannot remember the answers, remember to look more closely at that area during your revision.

A quantitative team at Orion Capital runs a machine learning–based global equity long–short strategy with occasional options overlays. The portfolio shows negative skewness and fat tails in historical returns. The team wants to improve its risk management and stress testing.

1. To estimate a 1‑day 99% VaR that captures the portfolio’s non‑normal and option‑driven payoffs, the most appropriate approach is to:
   1. Use a parametric variance–covariance VaR assuming multivariate normal returns.
   2. Use a historical simulation VaR using the last 3 months of daily returns.
   3. Use a Monte Carlo simulation with a multivariate normal distribution calibrated to 5 years of daily factor returns.
   4. Use a Monte Carlo simulation with a multivariate skewed t‑distribution calibrated to 5 years of daily factor returns.
2. The risk manager reports that the “1‑day 99% CVaR of the strategy is −4.5%.” This metric is best interpreted as:
   1. The maximum loss the portfolio can suffer in 1 day with 99% confidence.
   2. The average loss over all days, conditional on the loss exceeding the 99% VaR level.
   3. The minimum loss that will be exceeded with 1% probability over 1 day.
   4. The maximum drawdown that can occur over a 1‑day horizon with 99% confidence.
3. The team wants to understand the portfolio’s exposure to small changes in individual risk factors, such as interest rates, equity index levels, and FX rates, while holding other factors constant. The most appropriate tool is:
   1. Historical scenario analysis.
   2. Sensitivity analysis.
   3. Monte Carlo simulation of portfolio returns.
   4. Relative VaR versus the benchmark.
4. Orion’s backtest uses a 3‑year rolling window of historical returns to recalibrate the machine learning model and evaluate performance. Which limitation is most important from a risk standpoint?
   1. It will always overestimate volatility because the window is too short.
   2. It may fail to capture regime shifts and rare but severe market events not present in the rolling window.
   3. It double counts transaction costs in both training and testing periods.
   4. It makes VaR estimates impossible to compare across time.

Introduction

Simulation techniques, especially Monte Carlo simulation and scenario analysis, are core methods for modeling uncertainty and assessing risk in financial applications, particularly when combined with machine learning models. Understanding these approaches prepares you to interpret risk and return projections for portfolios under a range of possible market conditions, essential for robust investment management.

Machine learning–based strategies are often highly parameterized and non‑linear, with performance that can change markedly across market regimes. Simple historical performance metrics rarely capture their full risk profile. Simulation and scenario analysis allow you to:

• Generate large numbers of pseudo‑outcomes beyond the limited historical record.
• Explore tail events, regime shifts, and joint moves in multiple risk factors.
• Examine how model parameters, distributional assumptions, and dependence structures affect portfolio risk.

Key Term: Monte Carlo simulation
A method that uses random sampling from specified probability distributions to estimate the potential outcomes of complex processes by simulating many possible scenarios.

Key Term: scenario analysis
A technique for evaluating the impact of specific sets of conditions or changes in multiple risk factors on portfolio value or model outputs.

These tools are tested in Reading 37 (Measuring and Managing Market Risk) and Reading 38 (Backtesting and Simulation) and are frequently embedded in vignette‑style questions involving quantitative strategies.

Monte Carlo Simulation in Finance

Monte Carlo simulation is a computational tool that estimates potential results by generating thousands (or often tens of thousands) of random scenarios according to specified probability distributions. In finance, it is widely used for:

• Forecasting future portfolio values and return distributions.
• Estimating risk measures such as VaR and CVaR for complex portfolios.
• Stress testing risk models and capital allocations.
• Pricing complex derivatives or portfolios where closed‑form solutions are unavailable.
• Testing the robustness of machine learning–based strategies under a wide range of simulated market paths.

Monte Carlo is particularly valuable when:

• Payoffs are nonlinear (e.g., options, path‑dependent strategies, leveraged portfolios).
• The joint distribution of risk factors is non‑normal or exhibits tail dependence.
• Historical data are limited or may not be representative of future market regimes.

Simulation Design Steps

A well‑constructed Monte Carlo simulation involves several design decisions. On the exam, you are often asked to spot weaknesses in these steps.

1. Select the variable(s) of interest

   Define clearly what you want to simulate, for example:

   • Portfolio end‑of‑period value or return.
   • Distribution of strategy P&L over a 1‑day or 10‑day horizon.
   • Path‑dependent metrics such as maximum drawdown over 1 year.

2. Determine key decision and risk variables

   Identify the drivers of the simulated output, such as:

   • Asset or factor returns (e.g., equity indices, interest rates, credit spreads, FX rates).
   • Factor sensitivities (betas, durations, deltas) implied by your portfolio or ML model.
   • Portfolio weights, leverage, and rebalancing rules.

   For machine learning strategies, the “model output” (e.g., predicted returns or signals) may itself be an input into the simulation.

3. Choose the time horizon and number of simulation runs

   • Horizon must match the risk measure (e.g., 1‑day VaR vs 1‑year performance).
   • A larger number of simulation runs (e.g., 10,000+) improves the precision of tail estimates (1% or 5% quantiles) but increases computation time.

4. Assign probability distributions to inputs

   This is one of the most exam‑relevant steps.

   • Simple implementations assume normally distributed returns.
   • To reflect empirical features such as skewness and fat tails, you may instead use a Student’s t‑distribution or, more generally, a skewed t‑distribution.
   • When multiple assets or factors are involved, you should model a multivariate distribution to capture correlations and tail dependence.

   Calibrating these distributions typically uses a historical lookback period, but the analyst may combine short‑term and long‑term data to reflect current conditions.

5. Generate random samples for each trial

   For each simulation run:

   • Draw random shocks for each asset or factor from the chosen multivariate distribution.
   • Map these shocks through the portfolio or ML model (including any trading or rebalancing rules) to obtain portfolio returns or P&L.

   This step distinguishes Monte Carlo simulation from historical simulation, which re‑samples actual historical observations rather than draws from a parametric distribution (discussed later).

6. Aggregate and analyze simulated results

   Once all simulation runs are completed, calculate output metrics such as:

   • Mean and volatility of returns.
   • VaR and CVaR at various confidence levels.
   • Distribution plots and tail diagnostics.
   • Maximum drawdown and other path‑dependent statistics (if you simulate full paths).

These design choices directly influence the risk metrics produced. Exam questions often give you a brief description of a simulation and ask whether the setup is appropriate, or what biases it may introduce.

Historical Simulation versus Monte Carlo Simulation

Reading 38 distinguishes two major simulation approaches used to account for skewness, excess kurtosis, and tail dependence:

• Historical simulation

  • Randomly selects (with or without replacement, i.e., bootstrapping) from actual historical observations of asset or factor returns.
  • Preserves the empirical joint distribution, including nonlinear dependences and any fat tails present in the data.
  • Simple to implement, but entirely reliant on the assumption that the future will resemble the past. Rare events not in the historical sample cannot be generated.

• Monte Carlo simulation

  • Specifies a statistical distribution for each asset or factor and draws random values from that distribution.
  • Can incorporate flexible parametric forms (e.g., multivariate skewed t‑distribution) to model skewness, kurtosis, and tail dependence.
  • Requires more modeling choices and parameter estimation; more flexible but more prone to model misspecification.

On the exam, watch for:

• Historical simulation being criticized for not capturing unprecedented events or structural breaks.
• Monte Carlo being criticized when it assumes multivariate normality for portfolios known to have non‑normal or option‑like payoffs.

Worked Example 1.1

A portfolio manager wants to assess the possible 1‑year returns of an equity portfolio, assuming annual returns are normally distributed with a mean of 6% and a standard deviation of 12%. They plan to run 5,000 trial Monte Carlo simulations.

Answer:
The manager uses a random number generator to sample 5,000 annual returns from a normal distribution with mean 6% and standard deviation 12%. These simulated returns provide an empirical distribution from which risk metrics can be estimated.

For example, the 5% one‑year VaR can be approximated as the 5th percentile of the 5,000 simulated returns. Under the normal assumption, the theoretical 5% VaR in return space is:

$$\text{VaR}_{5\%} \approx -\big(0.06 - 1.65 \times 0.12\big) = 0.138 \text{ or } 13.8\%$$

meaning there is a 5% probability of losing at least 13.8% over one year. In practice, the manager would compute this value directly from the simulated distribution.

In a Level 2 question, you might be asked whether the normality assumption is reasonable (e.g., for an options portfolio it is not) or how changing the volatility input would affect the VaR estimate.

Scenario Analysis and Sensitivity Analysis

Scenario analysis differs from standard simulation in that it examines specific, plausible combinations of changes in risk factors or market conditions, often based on historic crises or hypothetical shocks, rather than random sampling. Sensitivity analysis isolates the effect of small changes in a single input factor to assess model or portfolio risk exposure.

Key Term: sensitivity analysis
Assessing how changes in a single variable affect a model’s output, holding other variables constant.

From the risk management reading:

• Sensitivity measures quantify “first‑order” exposure to a single risk factor (e.g., equity beta, bond duration, option delta).
• Scenario measures evaluate the effect on portfolio value of a set of changes in multiple risk factors (e.g., equity down 25%, credit spreads +300 bps, FX −10%).

Scenarios can be:

• Historical scenarios – re‑creating actual episodes such as the 2008 crisis, the 1987 crash, or a specific rate‑hike period.
• Hypothetical scenarios – forward‑looking sets of shocks that might be more extreme than any historical episode but are considered plausible.
• Stress tests – scenarios that are deliberately extreme and are used to examine solvency or survival; reverse stress tests work backwards from an unacceptable loss to identify what risk‑factor changes could cause it.

For machine learning–based strategies, scenario analysis is especially important because:

• The strategy may not have been live during a major crisis; scenario analysis can “replay” such periods.
• ML models can overfit normal regimes; scenarios help test robustness to structural breaks.

Worked Example 1.2

A risk analyst models the impact on a bond portfolio if interest rates immediately rise by 200 basis points and equity markets fall by 15%. This is tested using a scenario analysis within the existing model.

Answer:
The analyst specifies a joint shock: all relevant yield curves are shifted up by 2%, and equity indices are marked down by 15%. The portfolio model is then rerun using these shocked inputs:

• Bond prices are recomputed using higher yields, reflecting duration and convexity.
• Any equity holdings or equity‑linked positions are repriced with a 15% decline.
• Any machine learning strategy overlay that trades dynamically based on signals is evaluated as if these shocks occurred at once (or in a specified path).

The resulting change in portfolio value is the scenario loss. This provides concrete visibility into how adverse conditions could affect performance, complementing probabilistic VaR or Monte Carlo results that may not emphasize such extreme joint moves.

On the exam, scenario analysis is usually contrasted with VaR: scenario analysis does not attach a probability to the scenario; it simply tells you “if this happens, the portfolio loses X%.”

Interpreting Simulation Results

After running a Monte Carlo or historical simulation, or evaluating a set of scenarios, the outputs are assessed via several key risk measures.

• Average returns and volatility – summarize central tendency and dispersion but do not fully describe tail risk for non‑normal distributions.
• Value at Risk (VaR) – focuses on the left tail.

Key Term: value at risk (VaR)
The smallest dollar or percentage loss such that the probability of greater losses is at or below a specified probability level (e.g., 5% over 1 day or 1 month).

In simulated data, the $\alpha$‑percent VaR for losses (e.g., $\alpha = 5\%$) is found by:

• Sorting simulated returns from worst to best.

• Identifying the return at the $\alpha$‑percent quantile (e.g., the 500th worst in 10,000 simulations).

• Reporting VaR as the minimum loss exceeded with probability $\alpha$.

• Conditional VaR (CVaR) – often called expected shortfall.

Key Term: conditional value at risk (CVaR)
The expected loss, given that the loss exceeds the value at risk (VaR) threshold.

Mathematically:

In a simulation, CVaR is estimated simply as the average of the worst $\alpha$‑percent of simulated losses. Because Monte Carlo and historical simulation provide the entire distribution of outcomes, CVaR is easy to compute and is typically more informative about tail risk than VaR alone.

• Maximum drawdown

Key Term: maximum drawdown
The largest peak‑to‑trough percentage loss over a specified period, based on a time series or simulated path of portfolio values.

Maximum drawdown is particularly relevant for strategies with path‑dependent risks or significant leverage (e.g., many hedge funds and ML‑based trading strategies). To estimate it via simulation, you must simulate paths of portfolio values, not just independent end‑of‑period returns.

Monte Carlo, historical simulation, and scenario analysis highlight downside outliers and enable stress testing of investment strategies for extreme but plausible outcomes.

Worked Example 1.3

A Monte Carlo engine generates 10,000 1‑day P&L scenarios for a portfolio. Sorted from worst to best, the 1‑day percentage P&Ls for the 500 worst days range from −4.2% to −3.0%, with an average of −3.6%.

Answer:
Assuming 10,000 simulations, the 5% left tail corresponds to 500 observations.

• The 5% 1‑day VaR is approximately −3.0%, the 500th worst outcome (the minimum loss that occurs with 5% probability).
• The 5% 1‑day CVaR is the average of these 500 worst outcomes, i.e., −3.6%.

An exam vignette might ask you to distinguish between reporting −3.0% (VaR) and −3.6% (CVaR) and to explain why CVaR provides more information when losses are fat‑tailed.

Strengths and Practical Limitations

Monte Carlo and scenario analysis offer distinct advantages:

• Can model complex, nonlinear payoffs (options, path‑dependent strategies, leverage) where parametric VaR based on normality is unreliable.
• Provide visual and statistical diagnostic tools for highly uncertain environments (e.g., distributions of returns, loss quantiles, drawdowns).
• Allow explicit modeling of tail dependence and extreme joint moves via flexible multivariate distributions or custom scenarios.
• Facilitate robust testing of machine learning models by exposing them to synthetic regimes, including those not seen historically.

However, limitations are exam‑important:

• Dependence on distributional assumptions and parameters

  • Misspecified distributions (e.g., assuming multivariate normality for highly skewed returns) can severely underestimate tail risk.
  • Estimation error increases with the number of parameters (e.g., in a multivariate skewed t‑distribution you must estimate means, variances, correlations, skewness, and kurtosis).

• Calibration and lookback choices

  • Historical simulation and parameter calibration both rely on a lookback period. If the period omits major crises or regime shifts, the simulation will not reflect those risks.
  • Using only recent data may reflect current volatility but omit long‑term extremes; using very long data may mix regimes in a misleading way.

• Correlation and diversification assumptions

  • It is well known that correlations tend to spike during periods of financial stress. Simulations that assume constant correlations based on calm periods will overstate diversification benefits and understate potential losses.

• Rolling‑window backtesting issues

  • As highlighted in Reading 38, roll‑forward (walk‑forward) backtesting and historical simulation may fail to capture rare, unobserved, or regime‑changing events not found in historical records.
  • Survivorship bias, look‑ahead bias, and data snooping can all contaminate backtests and hence the calibration of simulations for machine learning strategies.

• Computational and model risk

  • Monte Carlo for large portfolios or complex ML strategies can be computationally intensive.
  • There is substantial model risk: different plausible distributional choices can produce materially different VaR and CVaR estimates.

Exam Warning

Monte Carlo and historical simulation results are only as reliable as the assumptions used for variable distributions, correlations, and parameter estimates. Exam questions often test your ability to:

• Recognize when a normality assumption is inappropriate (e.g., for portfolios with options or stop‑loss rules).
• Identify underestimation of risk due to correlations calibrated in tranquil periods.
• Evaluate backtests or simulations that ignore known structural breaks (e.g., pre‑ and post‑crisis regimes).

If a vignette describes a sophisticated simulation that nonetheless:

• Uses only a short, calm sample for calibration.
• Ignores known sources of nonlinear payoff.
• Or labels a stress test as “99% VaR,”

you should flag those as weaknesses.

Revision Tip

When comparing scenario and Monte Carlo results, always check whether:

• Tail risks (skewed, fat‑tailed loss events) are properly captured by the chosen distributions and lookback windows.
• Scenario analysis has covered extreme but plausible joint moves, including those not observed historically.
• For ML‑based strategies, simulations have considered the impact of model breakdown (e.g., signals failing in a new regime), not just market shocks.

Summary

Monte Carlo simulation and scenario analysis are companion tools essential for modeling uncertainty and managing risk in financial forecasts, especially for complex or machine learning–based strategies. Monte Carlo simulation models the full probability distribution of possible outcomes using random draws from assumed distributions, while historical simulation re‑samples actual historical data. Scenario analysis examines specific risks from prescribed shocks or crisis events and does not assign probabilities.

Interpretation of results requires attention to methodological choices and limitations: choice of distribution, calibration window, correlation assumptions, and the presence of nonlinear payoffs. VaR and CVaR extracted from simulated distributions quantify downside risk, while maximum drawdown captures path‑dependent vulnerability. Sensitivity and scenario analysis complement VaR by providing additional detail on exposures to individual risk factors and to extreme joint movements.

For the CFA Level 2 exam, you must be able to design, interpret, and evaluate these methods, recognizing common pitfalls such as overreliance on benign historical periods, mis‑specified distributions, and failure to account for regime shifts or extreme but plausible scenarios.

Key Point Checklist

This article has covered the following key knowledge points:

• The distinction between Monte Carlo simulation, historical simulation, and scenario analysis, and when each is most appropriate.
• The key design steps in a Monte Carlo simulation, including variable selection, distributional assumptions, and correlation modeling.
• How to estimate and interpret VaR, CVaR, and maximum drawdown from simulated output distributions.
• The role of sensitivity analysis and scenario analysis as additional tools to VaR, particularly for stress testing and reverse stress testing.
• Practical limitations of simulation methods, including parameter uncertainty, calibration window choice, and correlation behavior in crises.
• Common exam traps related to assuming normality, ignoring nonlinearity, and misinterpreting simulation output.

Key Terms and Concepts

• Monte Carlo simulation
• scenario analysis
• sensitivity analysis
• value at risk (VaR)
• conditional value at risk (CVaR)
• maximum drawdown