Sufficient Statistics and Exponential Families

Why This Matters

Every time you compute a sample mean and sample variance from Gaussian data, you are using sufficient statistics without realizing it. The sample mean captures all the information the data has about the population mean. You could throw away the original data points and lose nothing.

Sufficient statistics tell you when data compression is lossless for inference. Exponential families are the class of distributions where sufficient statistics take a particularly clean form. These two ideas together explain why so many classical estimators have the structure they do, and they underlie the theoretical guarantees for MLE in parametric models.

Mental Model

You observe $n$ data points and want to estimate $\theta$. A sufficient statistic $T(X)$ is a function of the data that captures everything the data can tell you about $\theta$. Given $T(X)$, the conditional distribution of the data does not depend on $\theta$. So $T(X)$ is a lossless summary for the purpose of inference.

The factorization theorem gives a simple test: the statistic $T(X)$ is sufficient if and only if the joint density factors into a piece that depends on $\theta$ only through $T$ and a piece that does not depend on $\theta$ at all.

Formal Setup and Notation

Let $X = (X_1, \ldots, X_n)$ be i.i.d. from $p(x | \theta)$ where $\theta \in \Theta$.

Definition

Sufficient Statistic

A statistic $T(X)$ is sufficient for $\theta$ if the conditional distribution of $X$ given $T(X)$ does not depend on $\theta$:

$p(X | T(X) = t, \theta) = p(X | T(X) = t) \quad \text{for all } \theta$

Equivalently, $T(X)$ captures all the information in $X$ about $\theta$. Once you know $T(X)$, the remaining randomness in $X$ is pure noise with respect to $\theta$.

Definition

Minimal Sufficient Statistic

A sufficient statistic $T$ is minimal sufficient if it is a function of every other sufficient statistic. That is, for any other sufficient statistic $U$, there exists a function $g$ such that $T = g(U)$. A minimal sufficient statistic achieves the maximum data reduction possible without losing information about $\theta$.

Main Theorems

Theorem

Neyman-Fisher Factorization Theorem

Statement

A statistic $T(X)$ is sufficient for $\theta$ if and only if the joint density (or pmf) can be factored as:

$p(x_1, \ldots, x_n | \theta) = g(T(x), \theta) \cdot h(x)$

where $g$ depends on the data only through $T(x)$, and $h$ depends on the data but not on $\theta$.

Intuition

The factorization says the likelihood splits into two parts. The part that depends on $\theta$ sees the data only through $T$. The part that depends on the full data does not care about $\theta$. So for the purpose of learning about $\theta$, $T$ is all you need.

Proof Sketch

(Sufficiency implies factorization): Write $p(x | \theta) = p(x | T(x), \theta) \cdot p(T(x) | \theta)$. Since $T$ is sufficient, $p(x | T(x), \theta) = p(x | T(x)) = h(x)$. Set $g(T(x), \theta) = p(T(x) | \theta)$.

(Factorization implies sufficiency): If $p(x | \theta) = g(T(x), \theta) \cdot h(x)$, then $p(x | T(x) = t, \theta) = p(x | \theta) / p(T(x) = t | \theta)$. The numerator is $g(t, \theta) h(x)$ and the denominator is $g(t, \theta) \sum_{x': T(x')=t} h(x')$. These cancel, giving $h(x) / \sum_{x': T(x')=t} h(x')$, which does not depend on $\theta$.

Why It Matters

The factorization theorem is the practical workhorse for finding sufficient statistics. You write down the likelihood, identify what functions of the data appear in the $\theta$-dependent part, and those functions form a sufficient statistic. For exponential families, this immediately identifies the natural sufficient statistics.

Failure Mode

The factorization must hold for ALL values of $\theta$ simultaneously. A common mistake is to find a factorization that works for one specific $\theta$ value but not all. Also, the factorization depends on the support of the distribution: if the support depends on $\theta$ (e.g., Uniform$(0, \theta)$), be careful with indicator functions.

Exponential Families

Definition

Exponential Family

A parametric family is an exponential family if the density can be written as:

$p(x | \theta) = h(x) \exp\!\left(\eta(\theta)^\top T(x) - A(\theta)\right)$

where:

• $T(x) \in \mathbb{R}^k$ is the sufficient statistic
• $\eta(\theta) \in \mathbb{R}^k$ is the natural parameter
• $A(\theta)$ is the log-partition function (ensures normalization)
• $h(x) \geq 0$ is the base measure

When the parameterization uses $\eta$ directly (i.e., $\eta$ is the free parameter), the family is in canonical form: $p(x | \eta) = h(x) \exp(\eta^\top T(x) - A(\eta))$.

Most distributions you encounter are exponential families: Gaussian, Bernoulli, Poisson, Exponential, Gamma, Beta, Multinomial, and Wishart. Notable exceptions: the Cauchy distribution, mixture models, and the Uniform$(0, \theta)$ distribution.

Key properties of exponential families:

1. Sufficient statistics: $T(X)$ is always sufficient (by factorization)
2. MLE is unique when it exists: the log-likelihood is concave in $\eta$ (strictly concave when the family is minimal and of full rank), so there are no local optima. Existence can fail at the boundary of the natural parameter space. Canonical failure cases: all-success or all-failure Bernoulli samples (MLE for $\eta = \text{logit}(p)$ is $\pm\infty$), all-zero Poisson samples ($\eta = \log\lambda = -\infty$), and separated data in logistic regression. Existence typically requires the observed sufficient statistic to lie in the interior of the convex hull of its support
3. Moment-generating properties: $\mathbb{E}[T(X)] = \nabla_\eta A(\eta)$ and $\text{Cov}(T(X)) = \nabla^2_\eta A(\eta)$. The log-partition function generates all the moments of $T$
4. Conjugate priors: every exponential family has a natural conjugate prior, making Bayesian inference tractable

Definition

Log-Partition Function

The log-partition function ensures normalization:

$A(\eta) = \log \int h(x) \exp(\eta^\top T(x)) \, dx$

It is always convex in $\eta$ (because it is a log of an integral of exponentials). Its first derivative gives the expected sufficient statistic: $\nabla A(\eta) = \mathbb{E}_\eta[T(X)]$. Its second derivative gives the variance: $\nabla^2 A(\eta) = \text{Cov}_\eta(T(X))$, which is the Fisher information in canonical form.

Completeness

Definition

Complete Statistic

A sufficient statistic $T$ is complete if for any function $g$:

$\mathbb{E}_\theta[g(T)] = 0 \text{ for all } \theta \implies g(T) = 0 \text{ a.s.}$

Completeness means there is no non-trivial function of $T$ that has mean zero for all $\theta$. In exponential families with natural parameter space containing an open set, the natural sufficient statistic is always complete.

Completeness matters because it guarantees uniqueness: if $T$ is complete and sufficient, then any unbiased estimator based on $T$ is the unique best unbiased estimator (UMVUE). This connects to the Rao-Blackwell theorem below.

Rao-Blackwell Theorem

Theorem

Rao-Blackwell Theorem

Statement

Let $U$ be any unbiased estimator of $\tau(\theta)$ and let $T$ be a sufficient statistic. Define:

$\tilde{U} = \mathbb{E}[U | T]$

Then $\tilde{U}$ is:

1. A function of $T$ alone (not of the full data)
2. Unbiased for $\tau(\theta)$
3. At least as good as $U$: $\text{Var}_\theta(\tilde{U}) \leq \text{Var}_\theta(U)$ for all $\theta$, with equality only if $U$ is already a function of $T$.

Intuition

Conditioning on a sufficient statistic can only help (or not hurt) estimation. The sufficient statistic contains all the information about $\theta$. Any remaining randomness in $U$ beyond what $T$ captures is pure noise. Conditioning on $T$ averages out this noise, reducing variance while preserving unbiasedness.

Proof Sketch

Unbiasedness: $\mathbb{E}[\tilde{U}] = \mathbb{E}[\mathbb{E}[U|T]] = \mathbb{E}[U] = \tau(\theta)$ by the tower property.

Variance reduction: By the law of total variance: $\text{Var}(U) = \mathbb{E}[\text{Var}(U|T)] + \text{Var}(\mathbb{E}[U|T]) = \mathbb{E}[\text{Var}(U|T)] + \text{Var}(\tilde{U})$.

Since $\mathbb{E}[\text{Var}(U|T)] \geq 0$, we get $\text{Var}(U) \geq \text{Var}(\tilde{U})$.

Why It Matters

Rao-Blackwell says: never ignore a sufficient statistic. If you have any unbiased estimator, you can improve it (or at least not hurt it) by conditioning on a sufficient statistic. Combined with completeness, this gives the Lehmann-Scheffe theorem: if $T$ is complete and sufficient, then $\mathbb{E}[U|T]$ is the unique minimum-variance unbiased estimator (UMVUE).

Failure Mode

Rao-Blackwell improves unbiased estimators, but unbiasedness itself is not always desirable. Biased estimators (like the James-Stein estimator or ridge regression) can have lower MSE. The Rao-Blackwell theorem operates within the class of unbiased estimators and cannot compare across that boundary.

Canonical Examples

Example

Sufficient statistic for Gaussian mean

Let $X_1, \ldots, X_n \sim \mathcal{N}(\mu, \sigma^2)$ with known $\sigma^2$. The joint density is:

$p(x|\mu) = (2\pi\sigma^2)^{-n/2} \exp\!\left(-\frac{1}{2\sigma^2}\sum_i(x_i - \mu)^2\right)$

Expanding the square: $\sum_i(x_i - \mu)^2 = \sum_i x_i^2 - 2\mu \sum_i x_i + n\mu^2$.

By factorization: $g(T, \mu) = \exp(-\frac{1}{2\sigma^2}(-2\mu n\bar{x} + n\mu^2))$ where $T = \bar{X} = \frac{1}{n}\sum_i X_i$. The sample mean is sufficient for $\mu$. This is an exponential family with natural parameter $\eta = \mu/\sigma^2$ and sufficient statistic $T = \sum_i x_i$.

Example

Exponential family form of the Poisson distribution

$p(x | \lambda) = \frac{\lambda^x e^{-\lambda}}{x!} = \frac{1}{x!} \exp(x \log\lambda - \lambda)$.

This is an exponential family with $T(x) = x$, $\eta = \log\lambda$, $A(\eta) = e^\eta = \lambda$, and $h(x) = 1/x!$. For $n$ i.i.d. observations, $T = \sum_i X_i$ is sufficient for $\lambda$.

Common Confusions

Watch Out

Sufficient does not mean minimal sufficient

The entire data vector $X = (X_1, \ldots, X_n)$ is always trivially sufficient (the identity is a sufficient statistic). The interesting question is how much you can compress. Minimal sufficiency gives the maximum compression. For exponential families with $k$-dimensional natural parameter, the minimal sufficient statistic is $k$-dimensional, regardless of sample size $n$.

Watch Out

Not all distributions are exponential families

Mixture distributions are not exponential families (the sufficient statistic dimension grows with $n$). The Cauchy distribution is not an exponential family. The Uniform$(0, \theta)$ is not (because the support depends on $\theta$). When you are outside exponential families, the clean theory of sufficient statistics and conjugate priors does not apply as neatly.

Summary

• A statistic $T(X)$ is sufficient if the conditional distribution of $X$ given $T$ does not depend on $\theta$
• Factorization theorem: $p(x|\theta) = g(T(x), \theta) \cdot h(x)$ characterizes sufficiency
• Exponential families: $p(x|\theta) = h(x) \exp(\eta(\theta)^\top T(x) - A(\theta))$
• The log-partition function $A(\eta)$ generates moments of $T$: $\mathbb{E}[T] = \nabla A$, $\text{Cov}(T) = \nabla^2 A$
• Completeness + sufficiency gives uniqueness of UMVUE
• Rao-Blackwell: condition on a sufficient statistic to improve any unbiased estimator

Exercises

ExerciseCore

Problem

Find the sufficient statistic for $\theta$ in the Bernoulli model: $X_1, \ldots, X_n \sim \text{Bernoulli}(\theta)$. Write the joint pmf in exponential family form and identify the natural parameter, sufficient statistic, and log-partition function.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Let $X_1, \ldots, X_n \sim \text{Uniform}(0, \theta)$. Show that $T = X_{(n)} = \max_i X_i$ is sufficient for $\theta$ but this is not an exponential family. Why does this matter for the MLE?

Hint

Reveal Solution

ExerciseResearch

Problem

Prove that in a $k$-parameter exponential family where the natural parameter space contains an open set, the natural sufficient statistic $T(X) = \sum_{i=1}^n T(X_i)$ is complete. Why does this, combined with Rao-Blackwell, imply that any unbiased estimator based on $T$ is UMVUE?

Hint

Reveal Solution

References

Canonical:

• Casella & Berger, Statistical Inference (2nd ed., 2002), Chapters 6-7
• Lehmann & Casella, Theory of Point Estimation (2nd ed., 1998), Chapters 1-4
• Keener, Theoretical Statistics (2010), Chapters 3-4

Current:

• Wasserman, All of Statistics (2004), Chapter 9

• Wainwright & Jordan, "Graphical Models, Exponential Families, and Variational Inference" (2008)

• van der Vaart, Asymptotic Statistics (1998), Chapters 2-8

Next Topics

Building on sufficient statistics and exponential families:

• Fisher information: the curvature of the log-likelihood, directly related to the log-partition function in exponential families
• Hypothesis testing for ML: using sufficient statistics to construct optimal tests
• EM algorithm: exploiting exponential family structure for latent variable models