Time-series analysis - Forecast evaluation instability and s...

Learning Outcomes

After reading this article, you will be able to identify and explain instability in time-series forecasts, assess forecasting models using out-of-sample performance metrics, and recognize and correct for seasonality within autoregressive models. You will understand how changing economic environments affect model reliability and apply techniques to detect and address seasonal patterns in financial data, key for CFA Level 2 success.

CFA Level 2 Syllabus

For CFA Level 2, you are required to understand key topics in time-series analysis as they relate to forecasting and model robustness. Focus your revision on the following areas:

• Identifying instability of time-series model coefficients and its implications for model reliability
• Comparing forecasting models using out-of-sample error metrics, including root mean squared error (RMSE)
• Distinguishing between in-sample and out-of-sample forecasts
• Detecting and adjusting for seasonality in time-series data and models
• Calculating and interpreting forecasts using autoregressive models that incorporate seasonal lags

Test Your Knowledge

Attempt these questions before reading this article. If you find some difficult or cannot remember the answers, remember to look more closely at that area during your revision.

1. What does instability of a time-series model's coefficients mean for forecast reliability?
2. How is out-of-sample forecast error typically evaluated and compared between competing time-series models?
3. For a series with strong quarterly seasonality, how should seasonality be tested and adjusted in an AR model?
4. What does it indicate if an AR(1) model’s residuals show significant correlation at lag 4 when using quarterly data?

Introduction

Time-series models are essential for financial forecasting, but their predictive accuracy can be compromised by coefficient instability and unmodeled seasonality. To ensure robust forecasts, analysts must assess whether a model’s parameters remain stable over time and evaluate forecast performance using appropriate error metrics. Additionally, many financial series exhibit distinct seasonal patterns that, if ignored, can bias results and limit forecast usefulness. This article addresses how to evaluate instability in forecasts and rigorously test for and incorporate seasonality in time-series models, reflecting key CFA Level 2 requirements.

Forecast Evaluation and Model Instability

When using time-series models to predict future values, it is critical to assess whether the relationship between the variables in the model is stable over time. Economic factors, regulatory changes, or structural market shifts can cause the relationships (i.e., the coefficients) in a model to change, reducing confidence in model-based forecasts.

Key Term: coefficient instability
The phenomenon where estimated model parameters change over different sample periods, undermining the reliability of out-of-sample forecasts.

Model instability is more likely when the sample period covers different economic regimes or after events that cause structural breaks in the data. Using too long a sample period increases the risk of including multiple regimes, while using too short a period reduces the statistical reliability of the estimates.

Worked Example 1.1

A bank uses quarterly GDP growth and overnight interest rates to forecast loan demand using data from 2000–2020. In 2008–2009, a major financial crisis occurred. What risk is associated with using the entire period to estimate model coefficients to forecast demand in 2024?

Answer:
The relationship between economic variables before and after the financial crisis may have shifted, resulting in different model coefficients across periods. Thus, using the entire sample risks instability, making forecasts less reliable for the current economic environment.

Comparing Forecasting Models: In-Sample vs. Out-of-Sample Accuracy

Evaluating a model’s predictive power requires testing against data not used in estimation—a process called out-of-sample forecasting. This helps determine if a model generalizes well, as in-sample fit does not equate to forecasting accuracy.

Key Term: in-sample forecast
Prediction for periods included in the data used to estimate the model’s parameters.

Key Term: out-of-sample forecast
Prediction for periods after (or outside) the sample used for model estimation, providing a realistic test of forecasting ability.

The standard criterion for comparing forecasting models is the root mean squared error (RMSE) calculated on out-of-sample forecast errors. The model with the lowest out-of-sample RMSE is considered superior for forecasting future values.

Key Term: root mean squared error (RMSE)
The square root of the average squared difference between forecasted and actual outcomes, typically applied to out-of-sample predictions.

Worked Example 1.2

Two AR models are developed to forecast monthly sales. Model A yields an out-of-sample RMSE of $84, while Model B's RMSE is $102. Which should be selected for future forecasting, and why?

Answer:
Model A is preferred since its out-of-sample RMSE is lower, indicating more accurate forecasting performance.

Exam Warning

Focusing only on in-sample fit metrics, such as R², can mislead selection of time-series models. Use out-of-sample RMSE for model comparison and forecast evaluation.

Interpreting Stability and Selecting the Forecast Period

A key decision in time-series analysis is the choice of sample period. A long period increases estimation reliability but raises the chance of coefficient instability; a short period improves stability but may lack sufficient data. Analysts must consider potential regulatory changes, structural breaks, or economic regime shifts when selecting an appropriate estimation window.

When a structural break or instability is detected (e.g., fundamental shifts in policy or economic relationships), it is often better to re-estimate the model using only the most recent, homogeneous period. Graphical analysis, rolling regression estimates, and formal statistical tests can be used to assess coefficient stability.

Worked Example 1.3

A central bank observes that a model estimated using 1995–2010 exchange rate data delivers poor forecasts after 2011, coinciding with the adoption of a new monetary policy. What should the analyst do to improve forecast accuracy?

Answer:
Re-estimate the model using data only from the post-2011 policy regime to improve coefficient stability and forecast reliability.

Seasonality in Time-Series Analysis

Many economic and financial data series display regular seasonal patterns—systematic fluctuations that repeat over fixed intervals (e.g., months, quarters). If seasonality is ignored, time-series forecasts may be biased or exhibit patterns in model residuals that violate assumptions.

Key Term: seasonality
Predictable periodic variation in a time series, occurring at fixed intervals (such as monthly, quarterly, or yearly).

Time-series models must be tested for seasonality by examining autocorrelations in residuals at seasonal lags (e.g., lag 12 for monthly data, lag 4 for quarterly data). Significant residual autocorrelation at a seasonal lag indicates that seasonality is present and the model is misspecified.

Key Term: seasonal lag
The lag corresponding to the frequency at which seasonality occurs (e.g., lag 4 for quarterly data, lag 12 for monthly data).

Worked Example 1.4

A retail chain models monthly revenue with an AR(1) model. The autocorrelation at lag 12 in residuals is sizable and statistically significant. What does this indicate, and what should be done?

Answer:
The significant autocorrelation at lag 12 suggests yearly seasonality not captured in the original model. The analyst should add the 12th seasonal lag (i.e., include $x_{t-12}$ as an explanatory variable) in the AR model to properly account for the seasonal pattern.

Correcting for Seasonality: Incorporating Seasonal Lags

If seasonality is detected, adjust the AR model by including an additional seasonal lag as an independent variable. For quarterly data, include $x_{t-4}$; for monthly, $x_{t-12}$. The general form becomes:

$y_t = b_0 + b_1 y_{t-1} + b_S y_{t-S} + \varepsilon_t$

where $S$ is the seasonal period.

Including the seasonal lag removes the residual autocorrelation due to seasonality, improving model specification and forecast accuracy.

Worked Example 1.5

A hotel uses an AR(1) model to forecast quarterly occupancy, but finds high residual autocorrelation at lag 4. After including $y_{t-4}$ as another regressor in the AR model, the residual autocorrelations are no longer significant. What has been achieved?

Answer:
Including the lag 4 (seasonal) term correctly models seasonality, resolves model misspecification, and enhances forecasting accuracy.

Revision Tip

Always test residuals of AR models for autocorrelation at the seasonal lag. If present, add the appropriate seasonal lag as an independent variable in your time-series model.

Forecasting with AR Models with Seasonal Lags

Once an AR model includes the correct seasonal lag, use the estimated coefficients and the most recent actual (or forecasted) values for both the immediate and seasonal lagged periods to calculate forecasts.

Worked Example 1.6

A quarterly AR model for a logistics company is estimated as:

$\ln y_t = 0.007 + 0.28 \ln y_{t-1} + 0.76 \ln y_{t-4}$

Given that last quarter's occupancy was 2,000 and occupancy four quarters ago was 1,500, compute the forecast for the next quarter (rounded to nearest integer).

Answer:
$\ln y_{t+1} = 0.007 + 0.28 \times \ln(2000) + 0.76 \times \ln(1500)$

$\ln(2000) \approx 7.601$, $\ln(1500) \approx 7.314$

$\ln y_{t+1} = 0.007 + 0.28 \times 7.601 + 0.76 \times 7.314 \approx 0.007 + 2.128 + 5.556 = 7.691$

$y_{t+1} = e^{7.691} \approx 2,194$

The forecast for the next quarter is approximately 2,194.

Summary

• Model instability means coefficients estimated from one time period may not accurately forecast another period due to changing economic or structural conditions. Always check for stability and use out-of-sample error to compare models.
• Evaluating forecasting accuracy requires the use of out-of-sample RMSE, not just in-sample fit statistics.
• Seasonality must be detected using autocorrelation tests at the appropriate lag and, if present, the model should include a suitable seasonal term.

Key Point Checklist

This article has covered the following key knowledge points:

• Explain why model coefficient stability is necessary for reliable time-series forecasts
• Evaluate and compare models using out-of-sample RMSE, reflecting true forecasting ability
• Detect, test for, and correct seasonality by examining autocorrelation at seasonal lags
• Adjust AR models to include seasonal lags when significant seasonal autocorrelation is detected
• Forecast using AR models with both non-seasonal and seasonal lags, applying most recent actual values

Key Terms and Concepts

• coefficient instability
• in-sample forecast
• out-of-sample forecast
• root mean squared error (RMSE)
• seasonality
• seasonal lag