Maximum Likelihood Estimation

Under regularity conditions (identifiability, compactness of $\Theta$, and continuity of $\log p(x|\theta)$ in $\theta$), the MLE is consistent:

$\hat{\theta}_{\text{MLE}} \xrightarrow{P} \theta^*$

as $n \to \infty$.

The normalized log-likelihood $\frac{1}{n}\ell(\theta) = \frac{1}{n}\sum_i \log p(x_i | \theta)$ converges to $\mathbb{E}_{\theta^*}[\log p(X | \theta)]$ by the law of large numbers. This expected value is maximized uniquely at $\theta = \theta^*$ by the information inequality (Gibbs' inequality): for any $\theta \neq \theta^*$,

$\mathbb{E}_{\theta^*}[\log p(X | \theta^*)] > \mathbb{E}_{\theta^*}[\log p(X | \theta)]$

because $D_{\text{KL}}(p_{\theta^*} \| p_\theta) > 0$. So the limiting objective has a unique maximum at $\theta^*$, and the maximizer of the empirical version converges to it.

Consistency is the minimal requirement for any estimator: with enough data, you recover the truth. MLE achieves this under mild conditions. Consistency requires the model to be identifiable: different parameters must give different distributions. If the model is overparameterized (as in neural networks), the MLE is not unique, and consistency of the parameter vector fails (though the fitted distribution may still converge).

Consistency can fail if: (1) the model is not identifiable (e.g., mixture models with label switching), (2) the parameter space is not compact and the MLE drifts to the boundary (e.g., variance estimate in a mixture with a component collapsing to a point), or (3) the number of parameters grows with $n$ (the "Neyman-Scott problem": with $n$ means and one variance, the MLE of the variance is inconsistent).

Under regularity conditions, the MLE is asymptotically normal:

$\sqrt{n}(\hat{\theta}_{\text{MLE}} - \theta^*) \xrightarrow{d} \mathcal{N}(0, I(\theta^*)^{-1})$

Equivalently, for large $n$:

$\hat{\theta}_{\text{MLE}} \approx \mathcal{N}\!\left(\theta^*, \frac{I(\theta^*)^{-1}}{n}\right)$

The variance $I(\theta^*)^{-1}/n$ is the Cramér-Rao lower bound, achieved by the MLE in the limit.

The log-likelihood $\ell(\theta)$ is a sum of i.i.d. terms. Near $\theta^*$, a Taylor expansion gives:

$\ell(\theta) \approx \ell(\theta^*) + \ell'(\theta^*)(\theta - \theta^*) + \frac{1}{2}\ell''(\theta^*)(\theta - \theta^*)^2$

Setting $\ell'(\hat{\theta}) = 0$ and solving: $\hat{\theta} \approx \theta^* - \ell'(\theta^*)/\ell''(\theta^*)$.

The numerator $\ell'(\theta^*) = \sum_i s(\theta^*; x_i)$ is a sum of i.i.d. mean-zero terms with variance $nI(\theta^*)$, so by the CLT: $\ell'(\theta^*)/\sqrt{n} \to \mathcal{N}(0, I(\theta^*))$.

The denominator $\ell''(\theta^*)/n \to -I(\theta^*)$ by the LLN.

Combining: $\sqrt{n}(\hat{\theta} - \theta^*) \approx \frac{-\ell'(\theta^*)/\sqrt{n}}{\ell''(\theta^*)/n} \to \frac{\mathcal{N}(0, I)}{-I} = \mathcal{N}(0, I^{-1})$.

Asymptotic normality is the basis for:

• Confidence intervals: $\hat{\theta} \pm z_{\alpha/2}/\sqrt{nI(\hat{\theta})}$
• Hypothesis tests: Wald test, likelihood ratio test, score test
• Model comparison: AIC, BIC (which penalize the number of parameters using the asymptotic distribution of the log-likelihood)

It also shows that MLE is asymptotically efficient: its variance matches the Cramér-Rao lower bound.

Asymptotic normality fails when: (1) $\theta^*$ is on the boundary of $\Theta$ (e.g., estimating a variance that could be zero), (2) the Fisher information is zero or degenerate (flat likelihood), (3) the model is misspecified (the asymptotic distribution changes. The "sandwich estimator" is needed), or (4) the sample size is too small for the approximation to be accurate (especially in high dimensions).

Let $T(X_1, \ldots, X_n)$ be any unbiased estimator of $\theta$ (i.e., $\mathbb{E}_\theta[T] = \theta$ for all $\theta$). Then:

$\text{Var}_\theta(T) \geq \frac{1}{n \, I(\theta)}$

In the multivariate case ($\theta \in \mathbb{R}^d$), the covariance matrix satisfies $\text{Cov}(T) \succeq (nI(\theta))^{-1}$ in the Loewner order.

An estimator that achieves equality is called efficient.

The Cramér-Rao bound quantifies the fundamental limit of estimation. No matter how clever your estimator, you cannot beat $1/(nI(\theta))$ in variance while remaining unbiased. The Fisher information $I(\theta)$ is the "price tag": more informative data (higher $I$) allows more precise estimation.

The bound comes from the Cauchy-Schwarz inequality applied to the inner product between the score $s(\theta; X)$ and the estimator $T(X)$ in the space of square-integrable functions. Since the score has mean zero and variance $I(\theta)$, and $\text{Cov}(T, s) = 1$ (by differentiating the unbiasedness condition), Cauchy-Schwarz gives $1 \leq \text{Var}(T) \cdot I(\theta)$.

The Cramér-Rao bound is the benchmark against which all estimators are measured. An estimator achieving the bound is called efficient: it extracts all the information in the data. The MLE is asymptotically efficient: as $n \to \infty$, its variance converges to $1/(nI(\theta))$.

The bound also reveals a deep connection between estimation and information: the minimum variance is the reciprocal of the Fisher information. This connects to the geometry of statistical models (information geometry), where $I(\theta)$ is the Riemannian metric on the parameter space.

The Cramér-Rao bound applies only to unbiased estimators. Biased estimators can sometimes achieve lower MSE than the Cramér-Rao bound (because $\text{MSE} = \text{Var} + \text{Bias}^2$, and a small increase in bias can yield a large decrease in variance). This is the bias-variance tradeoff, and it is why regularized estimators (ridge regression, LASSO) often outperform the MLE in high dimensions.

The James-Stein paradox makes this concrete: see below.