Moment Generating Functions

Why This Matters

The moment generating function is the main workhorse connecting light-tailed probability distributions to concentration inequalities. When the MGF exists in a neighborhood of zero (sub-Gaussian, sub-exponential, bounded), it powers the Chernoff method, exponential tilting, and the sub-Gaussian / sub-exponential machinery. When it does not exist (Cauchy, heavy-tailed power laws), the characteristic function is the strictly more general tool, and concentration results require different machinery (Chebyshev, truncation, Nemirovski-style norms).

The Chernoff method is: apply Markov's inequality to $e^{tX}$ and optimize over $t$. This is an MGF computation.

A random variable is sub-Gaussian with parameter $\sigma$ if and only if its MGF satisfies $M(t) \leq e^{\sigma^2 t^2 / 2}$. Within the sub-Gaussian world, the entire concentration story reduces to bounding the MGF.

Exponential families are distributions whose density is $\exp(\theta^T T(x) - A(\theta))$, relying on properties of the exponential function. The function $A(\theta)$ is the log-MGF of the sufficient statistic.

Core Definitions

Definition

Moment Generating Function

The moment generating function of a random variable $X$ is:

$M_X(t) = \mathbb{E}[e^{tX}]$

defined for all $t \in \mathbb{R}$ where this expectation is finite. The MGF may not exist for all $t$; when it exists in an open interval around 0, it determines the distribution uniquely.

Extracting Moments

The $k$-th moment of $X$ is the $k$-th derivative of $M_X$ at zero:

$\mathbb{E}[X^k] = M_X^{(k)}(0)$

This follows from differentiating under the expectation:

$M_X^{(k)}(t) = \mathbb{E}[X^k e^{tX}]$

and evaluating at $t = 0$. The interchange of differentiation and expectation is valid when $M_X$ exists in a neighborhood of $t$.

In particular: $\mathbb{E}[X] = M_X'(0)$ and $\mathbb{E}[X^2] = M_X''(0)$.

Main Theorems

Theorem

MGF Uniqueness Theorem

Statement

If two random variables $X$ and $Y$ have moment generating functions $M_X(t)$ and $M_Y(t)$ that are finite and equal for all $t$ in some open interval $(-\delta, \delta)$ with $\delta > 0$, then $X$ and $Y$ have the same distribution.

Intuition

The MGF encodes the entire distribution, not just the moments. If two distributions agree on their MGFs in a neighborhood of zero, they must be the same distribution. This is stronger than moment matching: there exist distinct distributions with identical moments of all orders, but they cannot have identical MGFs in a neighborhood of zero.

Proof Sketch

The MGF is related to the characteristic function $\varphi_X(t) = \mathbb{E}[e^{itX}]$ by analytic continuation. If $M_X(t)$ is finite on $(-\delta, \delta)$, the characteristic function extends analytically to a strip in the complex plane. By the uniqueness theorem for characteristic functions (Levy inversion), the distribution is determined.

Why It Matters

This theorem justifies the "MGF technique" for identifying distributions. If you compute the MGF of a sum of independent Gaussians and recognize it as the MGF of another Gaussian, you can conclude the sum is Gaussian. This approach is cleaner than convolution arguments.

Failure Mode

The MGF must exist in a neighborhood of 0, not just at 0 (where it always equals 1). Heavy-tailed distributions like the Cauchy distribution have no MGF. For those distributions, use the characteristic function instead, which always exists.

Proposition

MGF of Independent Sum

Statement

If $X$ and $Y$ are independent random variables whose MGFs exist, then:

$M_{X+Y}(t) = M_X(t) \cdot M_Y(t)$

Intuition

Independence means $\mathbb{E}[g(X)h(Y)] = \mathbb{E}[g(X)] \mathbb{E}[h(Y)]$. Apply this with $g(X) = e^{tX}$ and $h(Y) = e^{tY}$.

Proof Sketch

$M_{X+Y}(t) = \mathbb{E}[e^{t(X+Y)}] = \mathbb{E}[e^{tX} e^{tY}] = \mathbb{E}[e^{tX}] \mathbb{E}[e^{tY}] = M_X(t) M_Y(t)$, where the third equality uses independence.

Why It Matters

This is why MGFs are the natural tool for sums of independent variables. Addition of random variables corresponds to multiplication of MGFs. Taking logs: the cumulant generating function $\log M_X(t)$ is additive for independent sums.

Failure Mode

Fails without independence. For dependent variables, $\mathbb{E}[e^{tX}e^{tY}] \neq \mathbb{E}[e^{tX}]\mathbb{E}[e^{tY}]$ in general.

Canonical Examples

Example

MGF of a Gaussian

Let $X \sim N(\mu, \sigma^2)$. Then:

$M_X(t) = \mathbb{E}[e^{tX}] = \exp\left(\mu t + \frac{\sigma^2 t^2}{2}\right)$

This exists for all $t \in \mathbb{R}$. Setting $\mu = 0$: $M_X(t) = \exp(\sigma^2 t^2 / 2)$, which is the defining condition for sub-Gaussian random variables with parameter $\sigma$.

Example

The Chernoff method in one line

For any $t > 0$: $P(X \geq a) = P(e^{tX} \geq e^{ta}) \leq e^{-ta} M_X(t)$. The first step is monotonicity of $\exp$; the second is Markov's inequality. Optimize over $t > 0$ to get the tightest bound. This is the entire Chernoff method.

Common Confusions

Watch Out

Moments existing does not imply MGF exists

A distribution can have all moments finite yet have no MGF. The lognormal distribution has $\mathbb{E}[X^k] < \infty$ for all $k$ but $M_X(t) = \infty$ for all $t > 0$. The MGF is a stronger condition than having all moments.

Watch Out

MGF vs characteristic function vs cumulant generating function

The MGF is $M_X(t) = \mathbb{E}[e^{tX}]$. The characteristic function is $\varphi_X(t) = \mathbb{E}[e^{itX}]$, which always exists. The cumulant generating function (CGF) is $K_X(t) = \log M_X(t)$. The CGF is additive for independent sums. In concentration inequality proofs, you work with the CGF.

Exercises

ExerciseCore

Problem

Compute the MGF of a Bernoulli($p$) random variable. Use it to find $\mathbb{E}[X]$ and $\mathbb{E}[X^2]$.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Let $X_1, \ldots, X_n$ be i.i.d. $N(0, 1)$. Use MGFs to prove that $\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$ is distributed as $N(0, 1/n)$.

Hint

Reveal Solution

References

Canonical:

• Casella & Berger, Statistical Inference (2002), Chapter 2.3
• Billingsley, Probability and Measure (1995), Section 21

Current:

• Wainwright, High-Dimensional Statistics (2019), Chapters 2-3 (MGFs in concentration)
• Vershynin, High-Dimensional Probability (2018), Chapter 2 (sub-Gaussian MGF condition)