## **Learning Outcomes**

This article explains how backtesting and performance measurement are used to validate risk models and judge risk-taking results. After reading, you will be able to:

- Explain why backtesting is a core control for model risk and how it fits into model governance.
- Apply and interpret common risk-adjusted performance measures (Sharpe, Treynor, Jensen’s alpha, tracking error, information ratio, Sortino).
- Describe practical pitfalls in backtesting (data quality, regime change, and biased testing) and how firms reduce these issues.

## **GARP FRM Part 1 Syllabus**

For the GARP FRM Part 1 exam, you are expected to understand backtesting and performance measurement as part of risk data, models, and model risk, with a focus on the following syllabus points:

- Risk-adjusted performance measures and what risk each one uses (total risk, systematic risk, downside risk, and active risk).
- Model validation concepts, including comparing model outputs to realized outcomes (e.g., VaR exceptions) and the use of stress testing when models fail in crises.
- Data and governance foundations that support reliable testing (standardized data, audit review, and documented model changes).

## **Test Your Knowledge**

_Attempt these questions before reading this article. If you find some difficult or cannot remember the answers, remember to look more closely at that area during your revision._

1. A distinguishing feature of the Sharpe measure (relative to Treynor) is that it uses:
   a) beta of the portfolio  
   b) standard deviation of the portfolio return  
   c) tracking error relative to a benchmark  
   d) downside deviation relative to a minimum acceptable return

2. A portfolio has expected return 11%, benchmark expected return 9%, and tracking error 5%. What is the information ratio?

3. A bank backtests a 99% one-day VaR model over 250 trading days. Approximately how many VaR exceptions should it expect if the model is correct?

4. True/False (brief explanation required): “If a portfolio’s Treynor ratio is higher than the market’s but its Sharpe ratio is lower than the market’s, the portfolio is well-diversified.”

## **What Do You Know Already?**

- Can you explain the difference between validating a _model_ (e.g., VaR) and evaluating a _manager_ (risk-adjusted performance)?
- Do you know which performance measures use (i) total volatility, (ii) beta, (iii) downside deviation, and (iv) tracking error?
- When a model backtest looks “good,” can you list at least three reasons the result might still be misleading (data issues, biased sample, missing costs, regime change)?

Use these prompts to gauge your starting point; then focus your revision where your answers were least confident.

## **Introduction**

Risk management is not finished when a model produces a number (such as VaR) or when a portfolio reports a return. Firms must check whether models behave as expected and whether risk-taking is being rewarded appropriately. Two toolkits do this work:

- **Backtesting**, which compares model forecasts to realized outcomes.
- **Performance measurement**, which compares returns to the amount and type of risk taken.

These are closely linked to **model risk**. A model can be theoretically sound and still fail due to wrong assumptions, weak parameter estimates, or bad input data. The global financial crisis and case studies such as Long-Term Capital Management (LTCM) show that models calibrated in calm periods can break when correlations and volatility jump. Backtesting is one way to detect this mismatch early—provided the firm has reliable data and governance.

> **Key Term: Backtesting**  
> A validation method that compares model-generated forecasts (risk measures, prices, hedges, or strategy signals) to realized outcomes over a defined historical period to judge model accuracy and stability.

## **Backtesting in the model lifecycle and governance**

Backtesting sits inside a broader model governance process. At a minimum, a firm should be able to answer:

1. **What is the model used for?** (capital, limits, pricing, hedging, performance reporting)
2. **What data feeds it?** (positions, market data, reference data, mappings)
3. **How is it tested?** (backtests, benchmarking, stress testing, independent review)
4. **What happens when it fails?** (model change, limit reduction, overlays, more capital)

Backtesting is not only a quantitative exercise; it is also a governance exercise. Internal audit and independent review functions are often expected to evaluate monitoring procedures, confirm the reliability of risk information systems, and review assumptions (e.g., volatilities and correlations). They may also validate risk measurement by backtesting investment strategies and reviewing the integrity of the data used to compute risk metrics such as VaR.

> **Key Term: Model risk**  
> The risk of loss caused by using a model that is incorrect or misused. Common sources include input risk (bad data), estimation risk (poor parameters), valuation risk (pricing error), and hedging risk (hedges behave differently than the model predicts).

### A practical backtesting workflow

A defensible backtest is designed, not improvised. Typical steps include:

- **Define the object being tested**
  - a risk forecast (VaR/ES),
  - a valuation model (derivative price),
  - a hedge (hedge ratio and residual P&L),
  - or a trading strategy (signal-based returns).

- **Specify the horizon and sampling**
  - One-day vs ten-day horizons (important for liquidity and unwind assumptions).
  - Rolling windows vs fixed windows.
  - Use of calm periods _and_ stress periods.

- **Define the realized outcome**
  - For VaR: realized trading P&L over the horizon.
  - For hedges: hedged P&L vs unhedged P&L; also residual risk.
  - For strategies: net-of-cost returns (include fees, bid-ask spread, funding).

- **Set pass/fail criteria**
  - “Number of exceptions” for VaR (plus tests for clustering).
  - “Tracking error and information ratio” for active management.
  - “Hedge effectiveness” thresholds for hedging programs.

- **Document and escalate**
  - Who reviews results?
  - What actions follow repeated failures?
  - How are model changes approved?

### Data quality as a prerequisite

Backtesting is only as good as the data used to compute “realized outcomes.” Weak risk data aggregation can cause:

- missing positions (incomplete P&L),
- wrong entity/counterparty identifiers (misstated concentrations),
- inconsistent market data histories (false signals of model failure or success),
- manual workarounds that bypass controls.

Regulatory thinking (e.g., BCBS 239) treats data architecture, standard naming conventions, metadata, accuracy, completeness, and timeliness as core requirements because poor data can directly create model risk—especially input risk.

## **Backtesting risk models: VaR exceptions, stress testing, and hedge tests**

### VaR backtesting: the exception count

A common backtesting question is: _If the model says “loss should exceed VaR only 1% of days,” does that happen in practice?_

For a 99% one-day VaR, an **exception** (also called a breach) occurs when:

$$\text{Realized Loss}_{t} > \text{VaR}_{0.99,t}$$

If the model is correct and assumptions hold, exceptions should be rare and approximately consistent with the VaR confidence level.

> **Key Term: VaR exception (breach)**  
> A day on which the realized loss over the VaR horizon exceeds the model’s VaR forecast at the chosen confidence level.

Expected exceptions over $N$ days at confidence level $c$ are:

$$E[\text{exceptions}] = N(1-c)$$

So, for $N=250$ and $c=0.99$, expected exceptions are $250 \times 0.01 = 2.5$ (roughly 2–3 per year).

Backtesting goes beyond the expected value. Two frequent diagnostics are:

- **Unconditional coverage**: Is the total number of exceptions consistent with 1%?
- **Independence (clustering)**: Do exceptions occur randomly, or cluster during volatility spikes?

Clustering is common when a model uses constant volatility or slow-moving parameters.

### Worked Example 1.1: VaR exception backtest (count-based)

A bank backtests a 99% one-day VaR model over 250 trading days and observes 7 exceptions.

1. What number of exceptions is expected under a correct model?
2. How should a risk manager interpret 7 exceptions?

> **Answer:**
>
> 1. Expected exceptions:
>    $$E = 250(1-0.99) = 2.5$$
> 2. Observing 7 exceptions is materially above 2.5. A practical interpretation is:
>
> - the model may underestimate tail risk (volatility/correlation changes, fat tails),
> - positions may have nonlinear risk not captured (options, credit spread jumps),
> - the realized P&L may include costs/valuation effects not consistent with the VaR setup,
> - or data/P&L capture issues may exist.
>
> Many firms also map results to a “traffic-light” style framework (commonly used in practice for 99% VaR over 250 days): low exception counts suggest acceptable performance; higher counts trigger more scrutiny and possibly capital multipliers, overlays, or model redevelopment.

### Why stress testing matters (lesson from crises)

Quantile-based models calibrated in calm periods can fail during crises. Case studies highlight that correlations can jump and volatility can move far beyond historical estimates. When a model assumes stable relationships, it can understate losses precisely when the firm most needs accurate risk numbers. This is why regulators and firms emphasize **stress testing and scenario analysis** alongside probabilistic metrics such as VaR.

A practical point for FRM candidates: when you see statements like “25 standard deviation moves occurred multiple days in a row,” interpret this as evidence that the model distributional assumptions (often normality and stable variance) were not appropriate for crisis conditions.

### Backtesting hedges (hedging risk)

Hedging introduces its own model risk. A hedge that “works in the model” can disappoint in the market due to:

- basis risk (hedge instrument does not match exposure),
- parameter instability (betas, durations, correlations change),
- liquidity effects and transaction costs,
- nonlinearities (gamma/vega exposures) and discrete rebalancing.

A hedge backtest often measures the reduction in P&L volatility or tail loss:

- Compare unhedged vs hedged P&L distributions.
- Check how hedge effectiveness changes in stress windows.
- For dynamic hedges, include rebalancing frequency and costs.

## **Performance measurement: risk-adjusted returns**

Performance measurement asks a different question than VaR backtesting. Rather than “is the model accurate?”, it asks “was the realized (or expected) return adequate for the risk taken?” The FRM Part 1 curriculum emphasizes several widely used measures.

> **Key Term: Risk-adjusted performance measure**  
> A statistic that evaluates excess return relative to a specified notion of risk (total volatility, systematic risk, downside risk, or benchmark-relative risk).

### Sharpe ratio (total risk)

The Sharpe measure uses **total risk** (standard deviation):

$$\text{Sharpe} = \frac{R_P - R_F}{\sigma_P}$$

- Best suited when the portfolio is the investor’s complete portfolio or is well-diversified.
- Penalizes both upside and downside volatility.

### Treynor ratio (systematic risk)

The Treynor measure uses **systematic risk** (beta):

$$\text{Treynor} = \frac{R_P - R_F}{\beta_P}$$

- Interprets return per unit of market risk.
- Most meaningful when the portfolio is well-diversified so that idiosyncratic risk is small.

### Jensen’s alpha (abnormal return vs CAPM)

Jensen’s alpha measures performance relative to CAPM-required return:

$$\alpha_P = R_P - \left[R_F + \beta_P(R_M - R_F)\right]$$

- $\alpha_P>0$ suggests outperformance relative to CAPM (given beta).
- $\alpha_P<0$ suggests underperformance.

### Tracking error and information ratio (active management)

Tracking error is the standard deviation of active returns:

$$\text{Tracking Error} = \sigma(R_P - R_B)$$

The information ratio (IR) scales expected active return by active risk:

$$IR = \frac{E(R_P - R_B)}{\text{Tracking Error}}$$

> **Key Term: Tracking error**  
> The volatility of a portfolio’s returns relative to its benchmark, measured as the standard deviation of active returns $R_P - R_B$. It is the core risk input for the information ratio.

### Sortino ratio (downside risk)

The Sortino ratio uses downside deviation (returns below a minimum acceptable return $R_{\text{MIN}}$):

$$\text{Sortino} = \frac{R_P - R_{\text{MIN}}}{\text{Downside Deviation}}$$

This focuses risk measurement on “bad outcomes” relative to a target, rather than total volatility.

### Worked Example 2.1: Sharpe, Treynor, Jensen’s alpha, and IR

You are given:

- Portfolio expected return $R_P = 10\%$
- Risk-free rate $R_F = 3\%$
- Portfolio standard deviation $\sigma_P = 15\%$
- Portfolio beta $\beta_P = 0.8$
- Market expected return $R_M = 11\%$
- Benchmark expected return $R_B = 9\%$
- Tracking error $= 6\%$

Compute (i) Sharpe, (ii) Treynor, (iii) Jensen’s alpha, (iv) information ratio.

> **Answer:**
>
> **(i) Sharpe**
> $$\text{Sharpe}=\frac{0.10-0.03}{0.15}=\frac{0.07}{0.15}=0.4667$$
>
> **(ii) Treynor**
> $$\text{Treynor}=\frac{0.10-0.03}{0.8}=\frac{0.07}{0.8}=0.0875$$
>
> **(iii) Jensen’s alpha**  
> First compute the CAPM-required return:
> $$R_F+\beta_P(R_M-R_F)=0.03+0.8(0.11-0.03)=0.03+0.8(0.08)=0.094$$
> Then alpha:
> $$\alpha_P=0.10-0.094=0.006=0.6\%$$
>
> **(iv) Information ratio**  
> Expected active return:
> $$E(R_P-R_B)=0.10-0.09=0.01$$
> Therefore:
> $$IR=\frac{0.01}{0.06}=0.1667$$
>
> Interpretation (brief):
>
> - Positive alpha suggests outperformance vs CAPM given $\beta_P$.
> - IR is positive but modest, meaning benchmark outperformance is small relative to active risk.

### **Exam Warning**

> Common exam mistakes include:
>
> - Using the **wrong denominator**: Sharpe uses $\sigma_P$, Treynor and Jensen use $\beta_P$, IR uses tracking error, Sortino uses downside deviation.
> - Mixing **time units** (e.g., annual returns with monthly standard deviations) without scaling risk consistently.
> - Forgetting that Jensen’s alpha compares to **CAPM-required return**, not to the benchmark return (unless the benchmark is the market portfolio and CAPM assumptions are intended).

## **Backtesting, model risk, and reporting behavior in practice**

### Backtesting as a control for model risk categories

Backtesting can reveal several types of model risk:

- **Input risk**: If exceptions spike after a system change, the issue may be position capture, stale market data, or inconsistent identifiers across desks.
- **Estimation risk**: If the model performs poorly after volatility shifts, the parameter method may be too slow-moving or based on an overly calm window.
- **Valuation risk**: If backtested P&L includes large unexplained components, pricing models (especially for complex derivatives) may be inconsistent across front and back office.
- **Hedging risk**: If hedge backtests fail in stress windows, the hedge relies on relationships that do not hold in crises.

This ties directly to enterprise-level risk management: the board and senior management need risk reporting that is accurate, complete, timely, and clear, especially during stress periods when reporting frequency may change.

### Common backtesting pitfalls (and why they matter)

Even when the arithmetic is correct, backtests can mislead:

- **Biased samples**: Testing only calm periods can make tail models look accurate.
- **Overfitting (data snooping)**: Repeatedly changing a model until it fits history can produce poor future results.
- **Look-ahead bias**: Using information not available at the decision date (common in strategy tests).
- **Survivorship bias**: Using today’s index constituents to test past performance.
- **Ignoring costs and liquidity**: A strategy can look profitable before bid-ask spreads, funding costs, and market impact.
- **Regime change**: Correlations and volatilities can change sharply in crises, invalidating stable-parameter models.

A realistic control approach includes stress testing, sensitivity checks, independent review, and clear escalation rules when backtests deteriorate.

### Ethics and reporting: why “selective performance” is a red flag

Case studies of rogue trading and misleading reporting show that performance can be “managed” by selecting favorable time windows or hiding losses in obscure accounts. Sound governance expects managers and reviewers to look at strategy outcomes over appropriate horizons and to treat unusually smooth or consistently positive performance with skepticism—especially when it is inconsistent with the strategy’s stated risk.

## **Summary**

Backtesting is a discipline for comparing model forecasts to realized outcomes, used to detect model failure, poor assumptions, and data problems. It is part of a wider governance system that includes independent review, internal audit scrutiny, and clear reporting to senior management—supported by strong risk data aggregation and consistent standards.

Performance measurement complements backtesting by focusing on whether returns compensate for risk. FRM Part 1 emphasizes several core ratios (Sharpe, Treynor, Jensen’s alpha, information ratio, and Sortino), each matching excess return to a particular notion of risk (total, systematic, benchmark-relative, or downside). Correct interpretation depends on choosing the right measure for the portfolio context and using consistent inputs.

## **Key Point Checklist**

_This article has covered the following key knowledge points:_

- Backtesting compares model outputs to realized outcomes and is a core tool in model validation.
- Model risk includes input, estimation, valuation, and hedging risk; weak data quality is a direct driver of input risk.
- VaR backtesting often focuses on exception counts and whether exceptions cluster during volatility spikes.
- Stress testing and scenario analysis are necessary complements when models calibrated to calm periods fail in crises.
- Sharpe uses $\sigma_P$; Treynor and Jensen use $\beta_P$; IR uses tracking error; Sortino uses downside deviation.
- Governance and internal review matter: reliable backtesting requires controlled data, documented methods, and escalation rules.

## **Key Terms and Concepts**

- backtesting
- model risk (input, estimation, valuation, hedging risk)
- VaR exception (breach)
- Sharpe measure
- Treynor measure
- Jensen’s alpha
- tracking error
- information ratio
- Sortino ratio
- scenario analysis and stress testing
- risk data aggregation and reporting (data quality and governance)


Risk data, models, and model risk - Backtesting and performance measurement