Statistical Estimation

Cramér-Rao Bound

The fundamental lower bound on the variance of any unbiased estimator: no unbiased estimator can have variance smaller than the reciprocal of the Fisher information.

CoreTier 1Stable~50 min

Prerequisites

Fisher Information

Quiz (5)Pulse Check

Why This Matters

When you build an estimator, you want to know: how good can it possibly be? The Cramér-Rao bound answers this for unbiased estimators. It says the variance of any unbiased estimator of $\theta$ is at least $1/(nI(\theta))$ , where $I(\theta)$ is the Fisher information per observation. No cleverness in estimator design can beat this limit.

This bound tells you whether your estimator is efficient (achieves the bound) or whether there is room for improvement. It also connects estimation theory to information geometry: the Fisher information is the Riemannian metric on the statistical manifold.

Formal Setup

Let $X_1, \ldots, X_n$ be i.i.d. from $p(x|\theta)$ , where $\theta \in \Theta \subseteq \mathbb{R}$ .

Definition

Score Function

The score function is the derivative of the log-likelihood:

$s(x; \theta) = \frac{\partial}{\partial \theta} \log p(x|\theta)$

Key property: $\mathbb{E}_\theta[s(X; \theta)] = 0$ for all $\theta$ .

Definition

Fisher Information

The Fisher information is the variance of the score:

$I(\theta) = \text{Var}_\theta[s(X; \theta)] = \mathbb{E}_\theta[s(X; \theta)^2]$

Under regularity conditions, equivalently $I(\theta) = -\mathbb{E}_\theta[\partial^2 \log p(X|\theta) / \partial \theta^2]$ .

Definition

Efficient Estimator

An unbiased estimator $\hat{\theta}$ is efficient if it achieves the Cramér-Rao bound with equality: $\text{Var}_\theta(\hat{\theta}) = 1/(nI(\theta))$ for all $\theta$ .

Main Theorems

Theorem

Cramér-Rao Lower Bound

Statement

For any unbiased estimator $\hat{\theta}$ of $\theta$ :

$\text{Var}_\theta(\hat{\theta}) \geq \frac{1}{n I(\theta)}$

Equality holds if and only if the score function is a linear function of $\hat{\theta}$ : there exists $a(\theta)$ such that $\sum_{i=1}^n s(X_i; \theta) = a(\theta)(\hat{\theta} - \theta)$ a.s.

Intuition

The score function measures sensitivity of the likelihood to $\theta$ . High Fisher information means the data is very informative about $\theta$ , so estimators can be more precise. The bound quantifies this: more information means lower achievable variance.

Proof Sketch

Apply the Cauchy-Schwarz inequality to the covariance of $\hat{\theta}$ and the total score $S_n = \sum_{i=1}^n s(X_i; \theta)$ :

$[\text{Cov}(\hat{\theta}, S_n)]^2 \leq \text{Var}(\hat{\theta}) \cdot \text{Var}(S_n)$

Since $\hat{\theta}$ is unbiased, differentiating $\mathbb{E}[\hat{\theta}] = \theta$ under the integral gives $\text{Cov}(\hat{\theta}, S_n) = 1$ . Since $S_n$ is a sum of i.i.d. terms, $\text{Var}(S_n) = nI(\theta)$ . Substituting: $1 \leq \text{Var}(\hat{\theta}) \cdot nI(\theta)$ .

Why It Matters

This is the central result of classical estimation theory. It provides a universal benchmark: if your estimator's variance equals $1/(nI(\theta))$ , you know no unbiased estimator can do better. The MLE achieves this bound asymptotically, which is one reason MLE is the default estimation method.

Failure Mode

The bound only applies to unbiased estimators. Biased estimators can have lower MSE than the Cramér-Rao bound (this is the James-Stein phenomenon). The regularity conditions matter: for uniform distributions $U(0, \theta)$ , the support depends on $\theta$ , the bound does not apply, and the MLE converges at rate $1/n$ instead of $1/\sqrt{n}$ .

Theorem

Multivariate Cramér-Rao Bound

Statement

For any unbiased estimator $\hat{\theta}$ of $\theta \in \mathbb{R}^d$ :

$\text{Cov}_\theta(\hat{\theta}) \succeq \frac{1}{n} [I(\theta)]^{-1}$

where $\succeq$ denotes the Loewner (positive semidefinite) ordering and $I(\theta)$ is the $d \times d$ Fisher information matrix with entries $I_{jk}(\theta) = \mathbb{E}[\partial_j \log p \cdot \partial_k \log p]$ .

Intuition

The matrix inequality says that for any direction $v \in \mathbb{R}^d$ , the variance of $v^T \hat{\theta}$ is at least $v^T [nI(\theta)]^{-1} v$ . Parameters that are poorly identified (low Fisher information in their direction) have high minimum variance.

Proof Sketch

Same Cauchy-Schwarz argument applied to each component, or equivalently, consider $v^T \hat{\theta}$ for arbitrary direction $v$ and apply the scalar Cramér-Rao bound to this one-dimensional projection.

Why It Matters

In practice, models have multiple parameters. The matrix version tells you which parameters are easy to estimate (high Fisher information) and which are hard. It also connects to the natural gradient: gradient descent in Fisher information geometry converges in fewer steps because it accounts for parameter curvature.

Failure Mode

Same as the scalar case: applies only to unbiased estimators, requires regularity conditions, and biased estimators can achieve lower MSE.

Canonical Examples

Example

Estimating the mean of a normal distribution

Let $X_1, \ldots, X_n \sim N(\mu, \sigma^2)$ with $\sigma^2$ known. The Fisher information per observation is $I(\mu) = 1/\sigma^2$ . The Cramér-Rao bound gives $\text{Var}(\hat{\mu}) \geq \sigma^2/n$ . The sample mean $\bar{X}$ has variance exactly $\sigma^2/n$ , so it is efficient.

Example

Estimating the rate of a Poisson distribution

Let $X_1, \ldots, X_n \sim \text{Poisson}(\lambda)$ . The Fisher information is $I(\lambda) = 1/\lambda$ . The Cramér-Rao bound gives $\text{Var}(\hat{\lambda}) \geq \lambda/n$ . The sample mean $\bar{X}$ has variance $\lambda/n$ , so it is efficient.

Common Confusions

Watch Out

The Cramér-Rao bound does not apply to biased estimators

The bound $\text{Var} \geq 1/(nI)$ is only for unbiased estimators. A biased estimator can have lower variance. The bias-variance tradeoff means that the optimal MSE estimator is often biased. The James-Stein estimator beats $\bar{X}$ in MSE when the dimension $d \geq 3$ , despite $\bar{X}$ being efficient in the Cramér-Rao sense.

Watch Out

Achieving the bound does not mean achieving the best MSE

Efficiency (in the Cramér-Rao sense) means minimum variance among unbiased estimators. It does not mean minimum MSE among all estimators. For small samples, regularized or shrinkage estimators with some bias often have lower MSE than the efficient unbiased estimator.

Watch Out

Regularity conditions are not just technicalities

For $X \sim U(0, \theta)$ , the support depends on $\theta$ . The Cramér-Rao bound does not apply. The MLE $\hat{\theta} = X_{(n)}$ has variance of order $\theta^2/n^2$ , which is much smaller than $1/(nI)$ would suggest. When regularity fails, the parametric rate can be faster than $1/\sqrt{n}$ .

Summary

For unbiased estimators: $\text{Var}(\hat{\theta}) \geq 1/(nI(\theta))$
The proof is a single application of Cauchy-Schwarz
MLE is asymptotically efficient: it achieves the bound as $n \to \infty$
Biased estimators can beat the bound in MSE
Regularity conditions matter: the bound fails for non-regular families

Exercises

ExerciseCore

Problem

Compute the Fisher information for $X \sim \text{Bernoulli}(p)$ and state the Cramér-Rao bound for estimating $p$ from $n$ i.i.d. observations.

ExerciseAdvanced

Problem

Show that for the exponential family $p(x|\theta) = h(x)\exp(\eta(\theta) T(x) - A(\theta))$ , an efficient estimator exists if and only if $T(x)$ is a linear function of the natural parameter $\eta(\theta)$ .

Related Comparisons

Cramér-Rao Bound vs. Minimax Lower Bounds

References

Canonical:

Casella & Berger, Statistical Inference (2002), Chapter 7.3
Lehmann & Casella, Theory of Point Estimation (1998), Chapter 2
Schervish, Theory of Statistics (1995), Section 2.3 (information inequalities)

Current:

van der Vaart, Asymptotic Statistics (1998), Chapter 8
Cover & Thomas, Elements of Information Theory (2006), Chapter 11.10 (Fisher information and the Cramér-Rao bound)
Keener, Theoretical Statistics (2010), Chapter 3 (unbiased estimation and efficiency)

Next Topics

Asymptotic statistics: MLE achieves the Cramér-Rao bound asymptotically
Minimax lower bounds: going beyond unbiased estimators to minimax optimality

Last reviewed: April 2026

Prerequisites

Foundations this topic depends on.

Fisher InformationLayer 0B
Maximum Likelihood EstimationLayer 0B
Common Probability DistributionsLayer 0A
Sets, Functions, and RelationsLayer 0A
Basic Logic and Proof TechniquesLayer 0A
Differentiation in RnLayer 0A

Next Topics

Asymptotic StatisticsContinue →Minimax Lower BoundsContinue →