Asymptotic Statistics

If $\sqrt{n}(X_n - \mu) \xrightarrow{d} N(0, \sigma^2)$ and $g$ is differentiable at $\mu$ with $g'(\mu) \neq 0$, then:

$\sqrt{n}(g(X_n) - g(\mu)) \xrightarrow{d} N(0, \sigma^2 [g'(\mu)]^2)$

A smooth function of an approximately normal random variable is itself approximately normal. The variance gets multiplied by the squared derivative because $g$ locally looks like a linear function with slope $g'(\mu)$.

The delta method is the workhorse of applied statistics. Need the variance of $\log(\hat{p})$? Of $1/\bar{X}$? Of $\hat{p}_1 / \hat{p}_2$? The delta method gives you the answer immediately without simulation. It is also the basis for constructing confidence intervals on transformed parameters.

When $g'(\mu) = 0$, the first-order delta method gives a degenerate limit. You need the second-order delta method: if $g'(\mu) = 0$ and $g''(\mu) \neq 0$, then $n(g(X_n) - g(\mu)) \xrightarrow{d} \frac{1}{2}\sigma^2 g''(\mu) \chi^2_1$. The rate changes from $\sqrt{n}$ to $n$ and the limit is no longer normal.

If $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{p} c$ (a constant), then:

$X_n + Y_n \xrightarrow{d} X + c$

$X_n \cdot Y_n \xrightarrow{d} c \cdot X$

$X_n / Y_n \xrightarrow{d} X / c \quad \text{(when } c \neq 0\text{)}$

A sequence converging in probability to a constant behaves like a constant in the limit. You can add, multiply, or divide by it without disrupting distributional convergence. The key is that $Y_n$ must converge to a constant, not a random variable.

Slutsky's theorem is the glue that holds asymptotic arguments together. Every time you replace $\sigma$ with $\hat\sigma$ in a $z$-statistic and claim the limit is still standard normal, you are using Slutsky.

Slutsky fails if $Y_n$ converges in distribution to a non-degenerate random variable. In that case, you need joint convergence $(X_n, Y_n) \xrightarrow{d} (X, Y)$ and the continuous mapping theorem.

Under regularity conditions, the maximum likelihood estimator $\hat\theta_n$ satisfies:

$\sqrt{n}(\hat\theta_n - \theta_0) \xrightarrow{d} N(0, I(\theta_0)^{-1})$

where $I(\theta_0) = -\mathbb{E}[\nabla^2 \log f(X; \theta_0)]$ is the Fisher information matrix.

The MLE is approximately normal centered at the truth, with variance equal to the inverse Fisher information divided by $n$. This is the best variance any regular estimator can achieve (Cramér-Rao bound), so the MLE is asymptotically efficient.

This theorem justifies the standard practice of reporting MLE point estimates with standard errors computed from the observed Fisher information. It is the theoretical backbone of likelihood-based inference.

Regularity conditions fail at boundary parameters (e.g., variance = 0), non-identifiable models (e.g., mixture models with unknown number of components), and models where the support depends on the parameter (e.g., $\text{Uniform}(0, \theta)$). In these cases the MLE may have non-normal limits or converge at non-$\sqrt{n}$ rates.

Under regularity conditions, the log-likelihood ratio admits the expansion:

$\log \frac{L_n(\theta_0 + h/\sqrt{n})}{L_n(\theta_0)} = h^\top \Delta_n - \frac{1}{2} h^\top I(\theta_0) h + o_p(1)$

where $\Delta_n = \frac{1}{\sqrt{n}} \sum_{i=1}^n \nabla \log f(X_i; \theta_0) \xrightarrow{d} N(0, I(\theta_0))$.

At the local scale $h/\sqrt{n}$ around the truth, the statistical experiment looks like a Gaussian shift experiment. The sufficient statistic is $\Delta_n$ (the normalized score), and the Fisher information determines the signal strength. This is the deepest structural result in parametric statistics.

LAN shows that, asymptotically, all regular parametric problems reduce to Gaussian location problems. This unifies the theory of efficient estimation and optimal testing. It also implies the asymptotic minimax lower bound: no regular estimator can beat the Fisher information bound.

LAN fails for non-regular models where the Fisher information is zero or infinite, and for semiparametric or nonparametric models without finite-dimensional sufficient statistics.