Random Matrix Theory Overview

Let $X \in \mathbb{R}^{d \times n}$ have i.i.d. entries with mean 0 and variance $1/d$. Let $\hat{\Sigma} = X X^\top / n$ be the sample covariance. As $d, n \to \infty$ with $d/n \to \gamma > 0$, the empirical spectral distribution of $\hat{\Sigma}$ converges almost surely to the Marchenko-Pastur distribution with density:

$f_\gamma(\lambda) = \frac{1}{2\pi\gamma}\frac{\sqrt{(\lambda_+ - \lambda)(\lambda - \lambda_-)}}{\lambda} \cdot \mathbf{1}[\lambda_- \leq \lambda \leq \lambda_+]$

where $\lambda_{\pm} = (1 \pm \sqrt{\gamma})^2$.

When $\gamma > 1$, there is also a point mass of weight $(1 - 1/\gamma)$ at $\lambda = 0$, corresponding to the $d - n$ zero eigenvalues of the rank-deficient sample covariance.

Even when the true covariance is the identity ($\Sigma = I$), the sample eigenvalues do not cluster near 1. They spread out over the interval $[(1 - \sqrt{\gamma})^2, (1 + \sqrt{\gamma})^2]$. The larger $\gamma$ is, the wider this spread. At $\gamma = 1$, the lower edge touches 0. This is where the matrix becomes singular.

The Marchenko-Pastur law tells you: the bulk eigenvalue spread is a statistical artifact, not a signal. Eigenvalues inside the MP support are noise; eigenvalues outside it may be signal (this is the spiked model).

Marchenko-Pastur is the null model for high-dimensional spectral analysis. When you compute eigenvalues of a sample covariance and want to know which are "real" and which are noise, you compare against the MP distribution. Eigenvalues outside $[\lambda_-, \lambda_+]$ are potentially informative; eigenvalues inside are likely noise.

This directly impacts PCA: in high dimensions, you cannot just take the top eigenvalues of $\hat{\Sigma}$. You need to account for the MP bulk.

Marchenko-Pastur assumes i.i.d. entries and identity population covariance. For structured covariance (non-identity $\Sigma$), the limiting spectral distribution changes and is described by the generalized MP law. For non-i.i.d. entries (e.g., heavy-tailed), universality results show the bulk behavior persists but edge behavior may differ.

Let $W \in \mathbb{R}^{d \times d}$ be a symmetric matrix with i.i.d. upper-triangular entries having mean 0 and variance $1/d$. As $d \to \infty$, the ESD of $W$ converges to the Wigner semicircle distribution with density:

$f(x) = \frac{1}{2\pi}\sqrt{4 - x^2} \cdot \mathbf{1}[|x| \leq 2]$

The eigenvalues are supported on $[-2, 2]$ and follow a semicircular shape.

A random symmetric matrix has eigenvalues that fill out a semicircle. The spectral radius is $2$ (with high probability, the largest eigenvalue is near $2 + o(1)$). This is the simplest RMT result: Gaussian noise in a symmetric matrix gives semicircular eigenvalue distribution.

A spike $\theta > 0$ produces a detectable outlier eigenvalue in $\hat{\Sigma}$ if and only if $\theta > \sqrt{\gamma}$. When detected, the outlier eigenvalue converges to $\lambda = (1 + \theta)(1 + \gamma/\theta)$.

The corresponding eigenvector of $\hat{\Sigma}$ has non-trivial correlation with the true signal direction $v$ if and only if $\theta > \sqrt{\gamma}$. Below this threshold, the eigenvector is asymptotically orthogonal to $v$. PCA completely fails.

There is a sharp phase transition at $\theta = \sqrt{\gamma}$. Above it, PCA works (the top eigenvector of $\hat{\Sigma}$ aligns with the signal). Below it, PCA fails completely. The noise overwhelms the signal, and the top eigenvector of $\hat{\Sigma}$ is pure noise.

This threshold is higher when $\gamma$ is large (more dimensions relative to samples means you need a stronger signal to detect it).