Ridge Regression

Why This Matters

OLS is unbiased, but unbiasedness is not always desirable. When features are correlated, when $d$ is large relative to $n$, or when $X^\top X$ is ill-conditioned, OLS has high variance and often poor predictive performance. Ridge regression trades a small amount of bias for a large reduction in variance. And in terms of mean squared error, it can dominate OLS.

Ridge is the simplest regularization method, the one where everything has a closed form, and the entry point for understanding why regularization works.

Mental Model

OLS finds the minimum-norm solution to $y \approx Xw$. Ridge adds a penalty that pulls $w$ toward zero: among all $w$ that explain the data reasonably well, pick the one with small $\|w\|_2$. The penalty strength $\lambda$ controls the tradeoff. more penalty means smaller weights (less variance) but worse fit to training data (more bias).

Geometrically: OLS projects $y$ onto the column space of $X$. Ridge "shrinks" this projection, pulling $\hat{y}$ slightly toward the origin.

Formal Setup

We have the linear model $y = Xw^* + \varepsilon$ where $X \in \mathbb{R}^{n \times d}$, $y \in \mathbb{R}^n$, $\varepsilon \sim (0, \sigma^2 I)$.

Definition

Ridge Regression

The ridge estimator with regularization parameter $\lambda > 0$ minimizes:

$\hat{w}_{\text{ridge}} = \arg\min_{w \in \mathbb{R}^d} \|y - Xw\|_2^2 + \lambda \|w\|_2^2$

The objective is strictly convex (for $\lambda > 0$), so the solution is unique. Setting the gradient to zero:

$-2X^\top(y - Xw) + 2\lambda w = 0 \implies (X^\top X + \lambda I)w = X^\top y$

The closed-form solution is:

$\hat{w}_{\text{ridge}} = (X^\top X + \lambda I)^{-1} X^\top y$

The matrix $X^\top X + \lambda I$ is always invertible for $\lambda > 0$, even when $X^\top X$ is singular (e.g., when $n < d$).

SVD Interpretation

The SVD perspective is the cleanest way to understand what ridge does. Write the SVD of $X = U \Sigma V^\top$ where $U \in \mathbb{R}^{n \times r}$, $\Sigma = \text{diag}(\sigma_1, \ldots, \sigma_r)$, $V \in \mathbb{R}^{d \times r}$, and $r = \text{rank}(X)$.

Definition

Ridge Shrinkage Factors

In the SVD basis, the OLS fitted values are $\hat{y}_{\text{OLS}} = \sum_{j=1}^r u_j u_j^\top y$, a full projection onto each left singular direction. Ridge modifies this to:

$\hat{y}_{\text{ridge}} = \sum_{j=1}^r \frac{\sigma_j^2}{\sigma_j^2 + \lambda}\, u_j u_j^\top y$

The shrinkage factor for direction $j$ is:

$d_j(\lambda) = \frac{\sigma_j^2}{\sigma_j^2 + \lambda}$

This satisfies $0 < d_j(\lambda) < 1$. Directions with large singular values ($\sigma_j \gg \sqrt{\lambda}$) are barely shrunk: $d_j \approx 1$. Directions with small singular values ($\sigma_j \ll \sqrt{\lambda}$) are heavily shrunk: $d_j \approx 0$.

In the coefficient space: The ridge estimate in the $V$-basis is:

$V^\top \hat{w}_{\text{ridge}} = \text{diag}\!\left(\frac{\sigma_j}{\sigma_j^2 + \lambda}\right) U^\top y$

Compare to OLS: $V^\top \hat{w}_{\text{OLS}} = \text{diag}(1/\sigma_j) U^\top y$. Ridge replaces $1/\sigma_j$ with $\sigma_j/(\sigma_j^2 + \lambda)$, which is bounded even when $\sigma_j$ is small. This is why ridge stabilizes ill-conditioned problems.

Bias-Variance Tradeoff

Definition

Ridge Bias and Variance

Under the model $y = Xw^* + \varepsilon$ with $\varepsilon \sim (0, \sigma^2 I)$:

Bias: $\mathbb{E}[\hat{w}_{\text{ridge}}] - w^* = -\lambda(X^\top X + \lambda I)^{-1} w^*$

This is nonzero for $\lambda > 0$. ridge is biased. The bias increases with $\lambda$.

Variance: $\text{Var}(\hat{w}_{\text{ridge}}) = \sigma^2 (X^\top X + \lambda I)^{-1} X^\top X (X^\top X + \lambda I)^{-1}$

Compare to OLS variance: $\sigma^2 (X^\top X)^{-1}$. The ridge variance is smaller in the positive semidefinite sense.

MSE: $\text{MSE}(\hat{w}_{\text{ridge}}) = \text{Bias}^2 + \text{Variance}$. For some $\lambda > 0$, this is strictly less than $\text{MSE}(\hat{w}_{\text{OLS}})$.

Main Theorems

Theorem

Ridge Dominates OLS in MSE

Statement

Under the linear model $y = Xw^* + \varepsilon$ with $\mathbb{E}[\varepsilon] = 0$ and $\text{Var}(\varepsilon) = \sigma^2 I$, there always exists a $\lambda > 0$ such that:

$\text{MSE}(\hat{w}_{\text{ridge}}) < \text{MSE}(\hat{w}_{\text{OLS}})$

where $\text{MSE}(w) = \mathbb{E}[\|w - w^*\|_2^2]$.

More precisely, using the SVD, if $\sigma_1 \geq \cdots \geq \sigma_d > 0$ are the singular values of $X$ and $\theta_j = v_j^\top w^*$ are the true coefficients in the SVD basis, then:

$\text{MSE}(\hat{w}_{\text{ridge}}) = \sum_{j=1}^d \left[\frac{\lambda^2 \theta_j^2}{(\sigma_j^2 + \lambda)^2} + \frac{\sigma^2 \sigma_j^2}{(\sigma_j^2 + \lambda)^2}\right]$

The first term is the squared bias for direction $j$; the second is the variance.

Intuition

OLS is BLUE (Gauss-Markov), but "best unbiased" does not mean "best overall." By allowing a small bias, ridge dramatically reduces variance. The MSE tradeoff is favorable because the bias grows linearly in $\lambda$ (from zero) while the variance decreases. at $\lambda = 0$, the derivative of MSE with respect to $\lambda$ is always negative, so a small positive $\lambda$ always helps.

Proof Sketch

Write $\hat{w}_{\text{ridge}} = W_\lambda y$ where $W_\lambda = (X^\top X + \lambda I)^{-1} X^\top$. The MSE decomposes as:

$\text{MSE} = \|\text{Bias}\|_2^2 + \text{tr}(\text{Var}) = \lambda^2 w^{*\top}(X^\top X + \lambda I)^{-2} w^* + \sigma^2 \text{tr}\!\left[(X^\top X + \lambda I)^{-2} X^\top X\right]$

Diagonalize using the SVD to get the per-component formula. Differentiate with respect to $\lambda$ at $\lambda = 0$:

$\frac{d}{d\lambda}\text{MSE}\bigg|_{\lambda = 0} = \sum_j \left[\frac{-2\sigma^2}{\sigma_j^2}\right] + \text{(bias terms at } \lambda = 0 \text{ are zero)}$

This derivative is strictly negative, so MSE decreases for small $\lambda > 0$.

Why It Matters

This is the theoretical justification for regularization. Gauss-Markov says OLS is optimal among unbiased linear estimators. But the class of all linear estimators (including biased ones like ridge) contains estimators with strictly lower MSE. This is why regularization is not just a heuristic. It is provably beneficial.

Failure Mode

The theorem guarantees existence of a good $\lambda$ but does not tell you its value. In practice, $\lambda$ must be chosen by cross-validation or marginal likelihood maximization. The optimal $\lambda$ depends on the unknown $w^*$ and $\sigma^2$, so no fixed formula works universally.

Bayesian Interpretation

Ridge regression is equivalent to MAP estimation with a Gaussian prior.

Under the model: $w \sim \mathcal{N}(0, \tau^2 I), \quad y \mid X, w \sim \mathcal{N}(Xw, \sigma^2 I)$

The posterior is: $w \mid y, X \sim \mathcal{N}\!\left((X^\top X + \frac{\sigma^2}{\tau^2} I)^{-1} X^\top y, \;\sigma^2(X^\top X + \frac{\sigma^2}{\tau^2} I)^{-1}\right)$

The posterior mean (= MAP estimate, since the posterior is Gaussian) equals $\hat{w}_{\text{ridge}}$ with $\lambda = \sigma^2/\tau^2$.

Interpretation: Large $\tau^2$ means a weak prior (small $\lambda$, close to OLS). Small $\tau^2$ means a strong prior pulling $w$ toward zero (large $\lambda$, heavy shrinkage).

Choosing Lambda

Definition

Cross-Validation for Ridge

The standard approach: for each candidate $\lambda$ in a grid, compute the leave-one-out (LOO) or $K$-fold cross-validation error.

LOO shortcut: For ridge regression, the LOO error has a closed form:

$\text{CV}_{\text{LOO}} = \frac{1}{n} \sum_{i=1}^n \left(\frac{y_i - \hat{y}_i(\lambda)}{1 - h_{ii}(\lambda)}\right)^2$

where $h_{ii}(\lambda)$ is the $i$-th diagonal of the ridge hat matrix $H_\lambda = X(X^\top X + \lambda I)^{-1} X^\top$. This avoids refitting $n$ times.

Canonical Examples

Example

Ridge in the orthonormal design case

When $X^\top X = I$ (orthonormal columns), OLS gives $\hat{w}_{\text{OLS}} = X^\top y$ and ridge gives:

$\hat{w}_{\text{ridge}} = \frac{1}{1 + \lambda} X^\top y = \frac{1}{1 + \lambda} \hat{w}_{\text{OLS}}$

Each coefficient is multiplied by the same shrinkage factor $1/(1 + \lambda) < 1$. Ridge uniformly shrinks all coefficients toward zero. Compare this to lasso, which can set some coefficients exactly to zero.

Example

Ridge fixes multicollinearity

Suppose two features are nearly identical: $x_1 \approx x_2$. Then $X^\top X$ has a near-zero eigenvalue, and OLS assigns wildly different (large, opposite-signed) coefficients to $x_1$ and $x_2$.

Ridge adds $\lambda$ to every eigenvalue of $X^\top X$, stabilizing the inversion. The ridge coefficients for $x_1$ and $x_2$ are moderate and similar in magnitude, reflecting the fact that the data cannot distinguish between them.

Common Confusions

Watch Out

Ridge does NOT do variable selection

Ridge shrinks all coefficients toward zero but never sets any exactly to zero. If you need sparse models (some coefficients exactly zero), you need the lasso (L1 penalty), not ridge. Ridge is appropriate when you believe all features contribute, but some contribute less.

Watch Out

Lambda is not the prior variance

The Bayesian connection gives $\lambda = \sigma^2/\tau^2$, not $\lambda = \tau^2$. Large $\tau^2$ (diffuse prior, less regularization) corresponds to small $\lambda$. Students often confuse the direction.

Watch Out

Gauss-Markov does not contradict ridge being better

Gauss-Markov says OLS is the best linear unbiased estimator. Ridge is biased, so it falls outside the Gauss-Markov class. There is no contradiction in ridge having lower MSE than OLS. The theorem simply does not apply to biased estimators.

Summary

• Ridge objective: $\min_w \|y - Xw\|_2^2 + \lambda\|w\|_2^2$
• Closed form: $\hat{w} = (X^\top X + \lambda I)^{-1} X^\top y$
• SVD: ridge shrinks direction $j$ by factor $\sigma_j^2/(\sigma_j^2 + \lambda)$
• Always exists $\lambda > 0$ with lower MSE than OLS
• Bayesian: ridge = Gaussian prior on $w$ with $\lambda = \sigma^2/\tau^2$
• Does not do variable selection. all coefficients remain nonzero
• Choose $\lambda$ by cross-validation (LOO has a closed form shortcut)

Exercises

ExerciseCore

Problem

Show that for $\lambda > 0$, the matrix $X^\top X + \lambda I$ is always invertible, even when $X$ does not have full column rank.

Hint

Reveal Solution

ExerciseCore

Problem

Derive the bias of the ridge estimator. Show that $\mathbb{E}[\hat{w}_{\text{ridge}}] = (X^\top X + \lambda I)^{-1} X^\top X\, w^*$ and therefore the bias is $-\lambda(X^\top X + \lambda I)^{-1} w^*$.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Show that the derivative of $\text{MSE}(\hat{w}_{\text{ridge}})$ with respect to $\lambda$ is strictly negative at $\lambda = 0$ (assuming $w^* \neq 0$ and $\sigma^2 > 0$). This proves that a small amount of ridge regularization always helps.

Hint

Reveal Solution

Related Comparisons

• Ridge vs. Lasso Regression

References

Canonical:

• Hastie, Tibshirani, Friedman, Elements of Statistical Learning (2009), Chapter 3.4
• Hoerl & Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems" (1970)

Current:

• Murphy, Probabilistic Machine Learning: An Introduction (2022), Chapter 11.3
• Wainwright, High-Dimensional Statistics (2019), Chapter 7

Next Topics

Building on ridge regression:

• Lasso regression: L1 penalty for sparse variable selection
• Elastic net: combining L1 and L2 penalties
• Bias-variance tradeoff: the general principle at work