Regression Analysis
Regression is where descriptive statistics becomes modelling. Linear regression explains continuous outcomes with least squares. Logistic regression explains binary outcomes with probabilities, log-odds, and classification thresholds.
Linear vs logistic — why both matter
Linear regression (OLS)
Linear regression predicts a continuous outcome. It fits a line by minimising squared residuals, so coefficients are directly interpretable in the original scale.
If β₁ = 1.5, one-unit increase in X changes Y by 1.5 units on average, holding the rest fixed.
Logistic regression (MLE)
Logistic regression predicts a binary outcome. Instead of a straight-line prediction for Y itself, it models the log-odds and maps the result through a sigmoid into a probability.
That is why it is the standard tool for PD scorecards and default models: predictions stay naturally in the [0, 1] interval.
A useful order for learning regression
Start with the fitted relationship
Learn what the coefficient actually means before worrying about p-values or diagnostics.
Then study residuals
Residuals are the fingerprint of what the model failed to learn. Most serious modelling issues show up there first.
Then separate fit from inference
R², RMSE, coefficient significance, discrimination, and calibration are not the same dimension. A model can look good on one and weak on another.
Then connect it to validation
In risk work, the question is not only “does it fit?” but also “is it stable, interpretable, monotonic, and decision-useful?”
OLS in action — fit, residuals, and assumptions
Choose a data pattern and change noise level. The fitted line updates, the residual plot changes, and the assumption cards tell you where OLS looks comfortable and where it starts to break.
Logistic regression — probabilities, odds, and thresholds
Move the coefficients and threshold, then watch how the sigmoid, log-odds line, and confusion matrix respond. This is the cleanest way to feel why scorecards are built on logistic models.
Estimation, separation, and the link function
The interactive above lets you set β₀ and β₁ by hand. In reality those numbers are estimated from data — and the way they are estimated is exactly where some of the most common validation findings hide.
OLS has a formula; logistic does not
Linear regression solves in closed form: β = (XᵀX)⁻¹Xᵀy. Logistic regression has no such shortcut. It is fit by maximum likelihood, solved numerically (iteratively reweighted least squares / Newton-Raphson). The model maximises the probability of the observed default pattern.
Perfect & quasi-complete separation
If a variable (or combination) splits defaults from non-defaults perfectly, the likelihood keeps rising as the coefficient grows, so MLE pushes it toward ±∞. You see an enormous coefficient with an enormous standard error — a model that looks "perfect" in-sample and is useless out.
Common causes: a leakage variable, a category with zero defaults, or over-fine binning.
| Link function | Maps log-odds via | Shape | Where it shows up in risk |
|---|---|---|---|
| Logit (default) | p = 1 / (1 + e⁻ᶻ) | Symmetric, S-curve | PD scorecards — coefficients read directly as odds ratios (eᵝ), which is why it dominates |
| Probit | p = Φ(z) | Symmetric, thinner tails | Latent-variable / structural credit models; near-identical fit to logit in the middle |
| Complementary log-log | p = 1 − exp(−exp(z)) | Asymmetric | Rare events and discrete-time hazard — the bridge to survival / time-to-default (module 14) |
Score scaling — turning a coefficient into a scorecard
A PD model outputs a probability, but lenders operate on scores. Scaling is the linear map from log-odds to points: Score = Offset + Factor · ln(odds). Two policy choices fix it — the points to double the odds (PDO) and an anchor (a base score at a base odds). Move them and watch the whole score-to-PD relationship shift.
The five OLS assumptions and what failure looks like
| Assumption | What it means | How to check | What happens if it fails | Typical remedy |
|---|---|---|---|---|
| Linearity | E(Y|X) is linear in X | Residual vs fitted plot | Biased coefficients, systematic misspecification | Transform X, add polynomial / spline terms, change model class |
| Independence | Residuals are not serially correlated | Durbin-Watson, time structure review | Standard errors become misleading | Time-series correction, clustered / robust SE |
| Normality | Residuals are roughly Normal | QQ plot, residual skewness, Shapiro-Wilk | Inference weakens, especially in small samples | Transform Y, bootstrap, robust inference |
| Homoscedasticity | Residual variance is constant | Residual fan shape, Breusch-Pagan | SE and p-values become unreliable | Robust SE, weighted least squares, transform target |
| No multicollinearity | Predictors are not too strongly correlated | VIF, correlation matrix | Unstable coefficients, inflated uncertainty | Drop variables, combine variables, ridge / PCA |
Regression across the IRB and IFRS 9 stack
The same two regressions you just explored carry almost the entire risk-parameter estate. Knowing which model class maps to which parameter — and which rulebook governs it — is the difference between checking a metric and understanding what you are validating.
| Parameter | Typical model class | Why this class | Primary EU rulebook |
|---|---|---|---|
| PD | Logistic regression on WoE features | Binary default outcome, [0,1] output, odds-ratio interpretability, monotone scorecards | CRR3 · EBA/GL/2017/16 |
| LGD | Linear / fractional / Tobit regression, often two-stage (cure vs loss) | Continuous, bounded [0,1] severity; mass at 0 and 1 needs censored or two-part models | EBA/GL/2017/16 · downturn LGD EBA/GL/2019/03 |
| EAD / CCF | Linear / fractional regression | Continuous conversion factor, bounded, frequently floored | CRR3 · EBA/GL/2017/16 |
The IRB lens (Basel / CRR3)
For regulatory capital, PD/LGD/EAD feed risk-weighted assets. Estimates must carry a documented Margin of Conservatism for data and methodology weaknesses (EBA/GL/2017/16, MoC categories A, B and C). Under CRR3, in force across the EU from 1 January 2025, the output floor caps how far internal models can fall below the standardised approach — phasing from 50% in 2025 to 72.5% by 2030.
The IFRS 9 lens (accounting)
The same regressions are repurposed for expected credit loss: ECL ≈ PD × LGD × EAD, but point-in-time and forward-looking, not through-the-cycle. Stage 1 recognises 12-month ECL and Stage 2 lifetime ECL after a significant increase in credit risk (SICR). Change in lifetime PD is an important SICR indicator, but qualitative criteria and backstops also matter; SICR is not defined by one mechanical PD threshold.
What matters in practice
Why squared errors?
OLS punishes large misses more than small ones. That makes the math elegant and the estimator efficient under the right assumptions, but also sensitive to outliers.
Probability is not linear
Logistic regression works because it is linear in log-odds, not in raw probability. That is the key conceptual shift from linear regression.
Two different jobs
A logistic model can rank defaults well and still produce wrong PD levels. Good discrimination does not guarantee calibration.
Better fit is not always better model
In unpenalised least squares, adding predictors cannot worsen training RSS or raw R². Validation is where you discover whether that apparent improvement generalises or merely memorises noise.
Residuals are evidence
Residuals show whether the model systematically misses a region, a segment, or a structure. They are not decorative plots.
Why logistic dominates scorecards
Logistic + WoE gives monotonicity, interpretability, and a direct route from coefficients to score points and odds scaling.
What to leave this page with
Linear regression explains continuous outcomes through least squares. Logistic regression explains binary outcomes through log-odds and probability.
The useful order is: first understand the fitted relationship, then inspect residuals, then separate fit from inference, then connect the model to discrimination, calibration, and stability.
Once those pieces connect, regression stops being just coefficient reading and becomes a full model validation framework.
Primary references
Definitions and current regulatory statements were checked against official documentation. Library pages describe concepts and APIs; they are not substitutes for model-specific validation.