## **Learning Outcomes**

This article explains how Type I and Type II errors arise in hypothesis testing and why risk managers must manage the trade-off between false alarms and missed signals. After reading, you will be able to:

- Define Type I and Type II errors and interpret them in financial risk decisions
- Relate Type I and Type II errors to the significance level ($\alpha$) and test power ($1-\beta$)
- Explain how $\alpha$, sample size, and effect size jointly affect the probability of a Type II error and the power of a test

## **GARP FRM Part 1 Syllabus**

For the GARP FRM Part 1 exam, you are expected to understand hypothesis testing with a focus on the following syllabus points:

- Explain the difference between Type I and Type II errors and how these relate to the size and power of a test
- Understand how a hypothesis test and a confidence interval are related
- Explain the problem of multiple testing and how it can lead to biased results

## **Test Your Knowledge**

_Attempt these questions before reading this article. If you find some difficult or cannot remember the answers, remember to look more closely at that area during your revision._

1. In a hypothesis test, a Type I error occurs when you:  
   a) Reject a false null hypothesis  
   b) Fail to reject a false null hypothesis  
   c) Reject a true null hypothesis  
   d) Fail to reject a true null hypothesis

2. Short calculation: In an upper-tail z-test at $\alpha=5\%$, the critical value is approximately $z_{0.05}=1.65$. If your computed z-statistic is 1.19, what is your decision?

3. If you decrease the significance level from 5% to 1% while keeping sample size fixed, what happens to the probability of a Type II error and to the power of the test?

4. True/False (brief explanation required): “If we fail to reject $H_0$, we have accepted $H_0$ as true.”

## **What Do You Know Already?**

- Can you clearly describe (in words) what “rejecting $H_0$” means and what it does _not_ mean?
- Do you know which probability is controlled directly by choosing $\alpha$?
- Can you explain why risk managers sometimes prefer a higher $\alpha$ (more false alarms) rather than a lower $\alpha$?

Use these prompts to gauge your starting point; then focus your revision where your answers were least confident.

## **Introduction**

Hypothesis testing is a formal way to use sample data (such as historical returns, default rates, or model residuals) to make a decision about a population parameter (such as the true mean return, the true default probability, or the true regression coefficient). Because we only observe a sample, any decision can be wrong.

For financial risk management, the cost of being wrong is often the main story. A test that flags “model failure” too easily can create expensive false alarms (unnecessary hedging, model rebuilds, or capital allocation changes). A test that rarely flags problems can miss genuine deterioration (model drift, strategy decay, or rising credit losses). Type I and Type II errors provide the language for this trade-off.

> **Key Term: Null hypothesis ($H_0$)**  
> The benchmark statement about a population parameter that is assumed true for the purpose of the test and is the statement the test is designed to try to reject.

## **Decision outcomes and error types**

Every hypothesis test forces a binary decision:

- Reject $H_0$, or
- Fail to reject $H_0$ (not “accept $H_0$”).

But reality also has two states:

- $H_0$ is true, or
- $H_0$ is false.

That creates four possible outcomes.

| Decision \ True state |       $H_0$ true |      $H_0$ false |
| --------------------- | ---------------: | ---------------: |
| Do not reject $H_0$   | Correct decision |    Type II error |
| Reject $H_0$          |     Type I error | Correct decision |

The two errors are defined as follows.

> **Key Term: Type I error**  
> Rejecting the null hypothesis when it is actually true.
>
> **Key Term: Type II error**  
> Failing to reject the null hypothesis when it is actually false.

### Interpreting errors using “false positive” and “false negative”

It often helps to map hypothesis testing language to classification language:

- Type I error = false positive (you “detect” an effect that is not there)
- Type II error = false negative (you miss an effect that is there)

In risk settings, both can be costly:

- **Type I error (false alarm)** example: Concluding a trading strategy has positive alpha when the true alpha is zero, leading to capital allocation to noise.
- **Type II error (missed detection)** example: Concluding a model parameter is “not different from zero” when it is different from zero, missing a real sensitivity and leaving an exposure unhedged.

## **Significance level and control of Type I error**

In hypothesis testing, the analyst chooses a **significance level** before looking at the data. This choice sets the probability of a Type I error.

> **Key Term: Significance level ($\alpha$)**  
> The probability of a Type I error:
>
> $$
> \alpha = P(\text{reject }H_0 \mid H_0 \text{ true})
> $$

For example, $\alpha=0.05$ means that if $H_0$ is actually true, then (over repeated samples) you will incorrectly reject it about 5% of the time.

### Rejection regions and critical values

A test statistic is computed from the sample. For a population mean (with known population standard deviation), the z-test statistic is:

$$
z = \frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}}
$$

- $\bar{x}$ = sample mean
- $\mu_0$ = hypothesized mean under $H_0$
- $\sigma/\sqrt{n}$ = standard error of the sample mean

Given $\alpha$ and whether the test is one-tailed or two-tailed, we find a **critical value** from the distribution of the test statistic under $H_0$ and define a rejection region.

- **Upper-tail test**: reject when $z > z_\alpha$
- **Lower-tail test**: reject when $z < -z_\alpha$
- **Two-tailed test**: reject when $|z| > z_{\alpha/2}$

### Worked Example 1.1

A risk analyst tests whether the mean daily return of a strategy is greater than 0. Assume:

- Sample size $n=36$
- Sample mean $\bar{x}=0.50\%$
- Population standard deviation $\sigma=2.40\%$
- Significance level $\alpha=5\%$
- Upper-tail test ($H_A:\mu>0$)

Test:

$$
H_0:\mu \le 0 \quad\text{vs}\quad H_A:\mu>0
$$

> **Answer:**  
> **Step 1: Compute the standard error**
>
> $$
> SE(\bar{x})=\frac{\sigma}{\sqrt{n}}=\frac{2.40\%}{6}=0.40\%
> $$
>
> **Step 2: Compute the z-statistic**
>
> $$
> z=\frac{0.50\%-0}{0.40\%}=1.25
> $$
>
> **Step 3: Find the critical value**
>
> For an upper-tail test at $\alpha=0.05$, the critical value is approximately $z_{0.05}=1.65$.
>
> **Step 4: Decision**
>
> Because $1.25 < 1.65$, the statistic is not in the rejection region. **Fail to reject $H_0$**.
>
> **Interpretation in error terms**
>
> If the strategy truly has $\mu \le 0$, rejecting $H_0$ would be a Type I error. This test did not reject, so it did not commit a Type I error on this sample. But it could still be committing a Type II error if the true $\mu$ is actually positive.

## **Type II error and the power of a test**

While $\alpha$ is chosen by the analyst, the probability of a Type II error depends on several factors: the true (unknown) parameter, sample size, variability, and the chosen $\alpha$.

> **Key Term: Power of a test**  
> The probability of correctly rejecting $H_0$ when it is false:
>
> $$
> \text{Power} = P(\text{reject }H_0 \mid H_0 \text{ false}) = 1-\beta
> $$
>
> where $\beta = P(\text{Type II error})$.

Power answers a practical question risk managers often care about: **If a problem is real, how likely is the test to detect it?**

### Why Type II error is harder to compute

- $\alpha$ is computed under the assumption that $H_0$ is true.
- $\beta$ requires specifying what “$H_0$ is false” looks like (for example, what alternative mean $\mu_1$ is relevant).

In practice, power is evaluated at a particular alternative (or a range of alternatives). In model validation, for instance, you might ask: “If the true mean P&L drift is +5 bps/day, how likely is my monitoring test to flag it within a month?”

### Worked Example 2.1

A desk monitors the mean daily P&L of a hedged book. Assume daily P&L is approximately normal with known $\sigma = \$1.0$ million. You test:

$$
H_0:\mu = 0 \quad\text{vs}\quad H_A:\mu \neq 0
$$

using a two-tailed z-test at $\alpha=5\%$ with $n=25$ days. Compute the **Type II error probability** if the true mean is actually $\mu_1 = \$0.5$ million per day.

> **Answer:**  
> **Step 1: Standard error of the sample mean**
>
> $$
> SE(\bar{X})=\frac{\sigma}{\sqrt{n}}=\frac{1.0}{5}=0.2\ \text{million}
> $$
>
> **Step 2: Rejection thresholds in terms of $\bar{X}$**
>
> For a two-tailed test at $\alpha=0.05$, $z_{0.025}=1.96$.
>
> Reject $H_0$ when:
>
> $$
> \left|\frac{\bar{X}-0}{0.2}\right|>1.96
> \quad\Longleftrightarrow\quad
> |\bar{X}|>1.96(0.2)=0.392
> $$
>
> So we **fail to reject $H_0$** when:
>
> $$
> -0.392 \le \bar{X} \le 0.392
> $$
>
> **Step 3: Compute $\beta$ under the alternative $\mu_1=0.5$**
>
> Under the true mean $\mu_1=0.5$, the sampling distribution is:
>
> $$
> \bar{X} \sim N(0.5,\ 0.2^2)
> $$
>
> The Type II error probability is:
>
> $$
> \beta=P(-0.392 \le \bar{X} \le 0.392 \mid \mu=0.5)
> $$
>
> Convert to z-scores under the alternative:
>
> $$
> z_L=\frac{-0.392-0.5}{0.2}=\frac{-0.892}{0.2}=-4.46
> $$
>
> $$
> z_U=\frac{0.392-0.5}{0.2}=\frac{-0.108}{0.2}=-0.54
> $$
>
> Therefore:
>
> $$
> \beta=\Phi(-0.54)-\Phi(-4.46)
> $$
>
> Using common normal approximations: $\Phi(-0.54)\approx 0.2946$ and $\Phi(-4.46)\approx 0$ (very close).
>
> So:
>
> $$
> \beta \approx 0.2946
> $$
>
> **Step 4: Power**
>
> $$
> \text{Power} = 1-\beta \approx 1-0.2946=0.7054
> $$
>
> Interpretation: if the true mean is \$0.5 million/day, this 25-day test will detect it (reject $H_0$) about 71% of the time.

## **Trade-offs that matter for the FRM exam**

The FRM learning objective emphasizes how Type I and Type II errors relate to **size** and **power**, and how design choices shift the balance.

### Changing $\alpha$ changes the trade-off

Holding sample size fixed:

- Lowering $\alpha$ (for example, 5% to 1%) makes rejection harder.
  - Type I error probability decreases
  - Type II error probability increases
  - Power decreases

- Raising $\alpha$ makes rejection easier.
  - Type I error probability increases
  - Type II error probability decreases
  - Power increases

This is why “being conservative” with $\alpha$ (very small $\alpha$) is not free: it often means missing real effects.

### Sample size is the main tool to reduce Type II error without raising Type I error

For a fixed $\alpha$:

- Increasing $n$ reduces the standard error (for means, $SE=\sigma/\sqrt{n}$).
- The sampling distribution under the alternative becomes tighter around the true parameter.
- As a result, it becomes easier for the test statistic to cross the critical threshold when $H_0$ is false.
- Therefore, $\beta$ tends to fall and power rises.

This is a core idea in monitoring and backtesting: with more data, you can detect smaller effects without increasing false alarms.

### One-tailed vs two-tailed tests and power

For the same $\alpha$:

- A **one-tailed** test places all rejection probability in one tail, which lowers the critical value on that side (for example, 1.65 instead of 1.96 at 5%).
- If the true effect is in the hypothesized direction, a one-tailed test is more powerful.
- But it cannot detect effects in the opposite direction (because the rejection region is only on one side).

In risk terms, a one-tailed test is like installing an alarm that only triggers for one type of bad outcome.

### **Exam Warning**

> - The significance level $\alpha$ is **not** the probability that $H_0$ is true. It is $P(\text{reject }H_0 \mid H_0\ \text{true})$.
> - Do not say “accept $H_0$.” The correct language is **fail to reject $H_0$**.
> - If you decrease $\alpha$ with sample size fixed, power typically falls because the probability of a Type II error rises.

## **Applications in financial risk management**

Type I and Type II errors are not just exam definitions; they map directly to governance and model risk decisions.

### Strategy testing and multiple testing bias

When many strategies (or many factor signals) are tested on the same dataset, at least one can look statistically significant purely by chance. The nominal $\alpha$ is calibrated for a single test; repeated testing raises the chance of at least one Type I error.

Risk interpretation:

- Type I error here is “discovering” a profitable strategy that is actually noise.
- The outcome is often poor out-of-sample performance and model risk events.

Controls include limiting the number of tests, using holdout samples, and applying corrections (for example, family-wise error controls), but the exam-level point is: **multiple testing increases the probability of Type I errors relative to the stated $\alpha$ for one test**.

### VaR backtesting as a Type I vs Type II problem

In VaR backtesting, the null might be “the VaR model is correct.” Exceptions (breaches) are compared to what is expected under the model.

- Type I error: rejecting a well-calibrated VaR model (unnecessary remediation, increased capital add-ons, or model replacement)
- Type II error: failing to reject a bad VaR model (understated risk and insufficient capital)

Banks often design monitoring with attention to both outcomes because the costs are asymmetric.

### Regression model diagnostics and missed significance

In regression-based risk models (factor models, macro-credit models, yield curve regressions), a common test is whether a slope coefficient differs from zero.

A practical Type II error is: “concluding a variable is not statistically significant” when it is significant but the test has low power. For example:

- Multicollinearity can increase coefficient standard errors, making it more likely to fail to reject $H_0:\beta=0$ even when the true coefficient is not zero (a Type II error outcome).
- Heteroskedasticity can make standard errors unreliable, which can distort hypothesis testing decisions (either error type can rise because the test statistic is mismeasured).

### Monitoring thresholds and operational risk “false alarms”

Key risk indicators (KRIs) often use thresholds that trigger escalation. Although KRIs are not always framed as hypothesis tests, the logic is the same:

- Type I error: escalating an issue when the process is just showing normal variation
- Type II error: failing to escalate when a true control breakdown is underway

Choosing thresholds resembles choosing $\alpha$; increasing observation windows resembles increasing $n$.

## **Summary**

A hypothesis test can be wrong in two ways: rejecting a true null (Type I error) or failing to reject a false null (Type II error). The significance level $\alpha$ directly controls the Type I error probability, while the Type II error probability $\beta$ depends on $\alpha$, sample size, variability, and the true parameter under the alternative.

Power ($1-\beta$) measures how likely a test is to detect a real effect. Lowering $\alpha$ generally reduces false alarms but increases missed detections when sample size is fixed. Increasing sample size is the main way to raise power without increasing the Type I error rate.

## **Key Point Checklist**

_This article has covered the following key knowledge points:_

- Type I error is rejecting $H_0$ when $H_0$ is true; its probability is the significance level $\alpha$
- Type II error is failing to reject $H_0$ when $H_0$ is false; its probability is $\beta$
- Power equals $1-\beta$ and measures correct rejection when $H_0$ is false
- Decreasing $\alpha$ (with fixed $n$) tends to increase $\beta$ and reduce power
- Increasing sample size (with fixed $\alpha$) tends to reduce $\beta$ and increase power
- Multiple testing raises the chance of Type I errors relative to the stated single-test $\alpha$

## **Key Terms and Concepts**

- Null hypothesis ($H_0$)
- Alternative hypothesis ($H_A$)
- Type I error
- Type II error
- Significance level ($\alpha$)
- Power ($1-\beta$)
- Critical value
- Rejection region
- One-tailed test
- Two-tailed test
- Standard error
- Multiple hypothesis testing


Hypothesis testing - Type I and Type II errors