Likelihood-Ratio, Wald, and Score Tests

The likelihood-ratio statistic $\Lambda_n = -2\log\frac{\sup_{\theta\in\Theta_0}L_n(\theta)}{\sup_{\theta\in\Theta}L_n(\theta)} = 2[\ell_n(\hat\theta_n) - \ell_n(\tilde\theta_n)]$ satisfies $\Lambda_n\to_d\chi^2_q$ under $H_0$ as $n\to\infty$, where $q$ is the number of restrictions imposed by $H_0$.

$\Lambda_n$ is twice the height difference between the unconstrained and constrained log-likelihood. Under the null, the constraint is true and the height drop is small; expand the log-likelihood to second order around the unrestricted MLE and the drop is a quadratic form in the constraint, asymptotically Chi-squared.

The LRT is the natural test from the Neyman-Pearson perspective: it generalizes the simple-versus-simple most-powerful test to composite hypotheses. Under regularity, Wilks's theorem guarantees the asymptotic Chi-squared reference, and the test is asymptotically locally most powerful in many parametric models. The factor $-2$ exists so the leading-order quadratic form has the natural Chi-squared scaling.

Wilks's theorem fails at parameter-space boundaries (the asymptotic distribution becomes a mixture of Chi-squareds with different degrees of freedom; this is the Chernoff-1954 phenomenon). It also fails when the model is not regular (parameters on a boundary, identifiability failures, non-differentiable likelihoods). For boundary problems, see Self-Liang (1987). For non-regular problems, the LRT typically has a non-standard limiting distribution that must be derived case by case.

The Wald statistic is $W_n = n(\hat\psi_n - \psi_0)^\top \hat\Sigma_n^{-1}(\hat\psi_n - \psi_0),$ where $\hat\Sigma_n$ is a consistent estimator of the asymptotic covariance of $\sqrt n\hat\psi_n$ (typically the $\psi\psi$ block of the inverse observed or expected Fisher information at the unrestricted MLE). Under $H_0$, $W_n\to_d\chi^2_q$.

$W_n$ standardizes the unrestricted MLE $\hat\psi_n$ by its estimated standard error and squares the result. It is the multivariate $z^2$ statistic: how many standard deviations is the MLE from the null value, squared and summed across components. Under the null and large samples, the standardized MLE is approximately standard Normal, so its squared norm is approximately Chi-squared.

Most regression-output software (R's summary.lm, Python's statsmodels, Stata) reports Wald-style test statistics by default, because all that is needed is the MLE and its estimated covariance. The $p$-values associated with individual regression coefficients are Wald tests on each coefficient against zero. The Wald test is the basis for the standard confidence interval $\hat\psi_n\pm 1.96\cdot\hat{\text{SE}}$ in the scalar case.

The Wald test is not invariant under reparameterization. If you transform $\psi$ by a smooth nonlinear map $g$, the Wald statistic for $H_0: g(\psi) = g(\psi_0)$ can give a different $p$-value than the original Wald statistic for $H_0: \psi = \psi_0$, even though the hypotheses are equivalent. This is the Hauck-Donner phenomenon: when the unrestricted MLE is far from the null, the Wald statistic can shrink toward zero, paradoxically reducing rejection. The LRT does not have this problem.

The score statistic is $S_n = \frac{1}{n}U_n(\tilde\theta_n)^\top I_n(\tilde\theta_n)^{-1}U_n(\tilde\theta_n),$ where $U_n(\tilde\theta_n)$ is the score vector evaluated at the constrained MLE and $I_n(\tilde\theta_n)$ is the observed (or expected) Fisher information at the constrained MLE. Under $H_0$, $S_n\to_d\chi^2_q$.

Equivalently, $S_n = n\bar U^\top I^{-1}\bar U$ where $\bar U = (1/n)U_n(\tilde\theta_n)$ is the average score per observation evaluated under the null.

The score is the gradient of the log-likelihood. Under the null, the gradient at the constrained MLE is in directions orthogonal to the null parameter (where the constrained MLE has zero score by definition). The score test asks: is the gradient in the directions we constrained meaningfully nonzero? If yes, the null point is not a good fit; reject. The statistic is the squared norm of the relevant score components, weighted by the inverse Fisher information.

The score test is computed entirely from the null fit; no alternative-model fit is required. This makes it the test of choice when the alternative is expensive (e.g., when testing whether a regression coefficient should be added: fitting the simpler model is cheap, and the score test for adding a variable uses only that fit). The score test is also the basis of the Pearson Chi-squared test: the Pearson statistic is the score test for goodness of fit against the unrestricted multinomial.

The score test is invariant under reparameterization of the parameter being tested, but not under reparameterization of the nuisance parameter. It can have low power when the alternative implies a particular non-standard score behavior. The asymptotic Chi-squared is also less accurate in small samples than the LRT for many models; the score test can have inflated or deflated Type I error for moderate $n$. Adjustments (Edgeworth expansion, bartlett correction) exist but are model-specific.

Under a regular parametric model and contiguous local alternatives $H_n: \psi = \psi_0 + h/\sqrt n$, the three statistics $\Lambda_n$, $W_n$, $S_n$ all converge in distribution to the same noncentral Chi-squared $\chi^2_q(h^\top[I^{-1}]_{\psi\psi}^{-1}h)$. Under the null ($h = 0$), the common limit is the central $\chi^2_q$. The three statistics agree to first order under the null: $|\Lambda_n - W_n| = O_p(n^{-1/2})$ and $|\Lambda_n - S_n| = O_p(n^{-1/2})$.

All three are second-order approximations to the same local log-likelihood quadratic form. They differ in which side of the maximum they expand from (LRT compares both sides; Wald expands from the unrestricted MLE; score expands from the constrained MLE), but the second-order term is the same to leading order, so the asymptotic distributions agree.

The equivalence is what justifies any of the three as "the" asymptotic test for the same null. In practice, choose by computational convenience. For small samples, the agreement breaks down, and the LRT typically has the best small-sample size properties; the Wald test has the worst (Hauck-Donner) when the true parameter is far from the null. Empirical studies (Mantel-Haenszel for binary data, mixed-effects models, generalized linear models) consistently favor the LRT or score test over Wald for moderate $n$.

The equivalence requires regularity. At boundary nulls (the variance of a random effect being zero), the trinity diverges: the LRT has a mixture-of-Chi-squareds limit, the Wald test has an asymmetric standard error, and the score test has a one-sided limit. None of the standard $p$-values are correct in that case.