Chi-Squared Distribution and Tests

If $Z_1,\dots,Z_k$ are i.i.d. $\mathcal{N}(0,1)$, then $\sum_{i=1}^k Z_i^2 \sim \chi^2_k.$ For $a\in\mathbb{R}$, $aZ_1$ is Normal with variance $a^2$, so $(aZ_1)^2/a^2 = Z_1^2\sim\chi^2_1$; summing $k$ such terms gives the result.

Squaring a standard Normal kills its sign and gives a positive random variable with density proportional to $z^{-1}e^{-z/2}/\sqrt z$, which is the $\chi^2_1$ density. Summing $k$ independent squared Normals adds the shape parameter (Gamma additivity).

Every classical sampling distribution involving sums of squared deviations from a Normal model is Chi-squared. The sample variance of an i.i.d. Normal sample has $(n-1)S^2/\sigma^2\sim\chi^2_{n-1}$. The residual sum of squares in linear regression with Normal errors is a Chi-squared with degrees of freedom equal to $n$ minus the number of regression coefficients.

The result requires independence and unit-variance Normality. For non-Normal samples, $\sum Z_i^2$ is not Chi-squared; finite-sample tests that assume Chi-squared on quadratic forms of non-Normal data are misspecified. The asymptotic version (Pearson Chi-squared below) makes a weaker assumption and recovers Chi-squared in the limit.

Let $X_1,\dots,X_n$ be i.i.d. $\mathcal{N}(\mu,\sigma^2)$ and let $S^2 = (1/(n-1))\sum(X_i-\bar X_n)^2$ be the unbiased sample variance. Then $\frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1},$ and the sample mean $\bar X_n$ is independent of $S^2$.

The sample mean is the projection of the sample vector onto the all-ones direction. The deviations $X_i - \bar X_n$ live in the orthogonal hyperplane, a space of dimension $n - 1$. Their squared norm, scaled by $\sigma^2$, is the squared length of an $(n-1)$-dimensional standard Normal vector, which is $\chi^2_{n-1}$. The independence of mean and variance is the orthogonality.

This is the building block of the Student-t and F sampling distributions. The t-statistic $(\bar X_n - \mu)/(S/\sqrt n)$ is a ratio of a Normal and a root-Chi-squared (over its degrees of freedom), so it is exactly $t_{n-1}$. The F statistic in F distribution and ANOVA is a ratio of two scaled sample variances, so it is exactly $F_{d_1,d_2}$.

The exact Chi-squared distribution of $S^2$ and the independence of $\bar X_n$ from $S^2$ are special properties of the Normal distribution. For non-Normal i.i.d. samples, $S^2$ is asymptotically normal with variance involving the fourth central moment, and is only asymptotically uncorrelated with $\bar X_n$, not independent in finite samples. The exact t-distribution argument therefore fails outside the Normal model.

Let $O_1,\dots,O_k$ be the observed counts of $n$ i.i.d. observations falling into $k$ cells, and let $E_i = np_i$ be the expected counts under the null hypothesis $p_1,\dots,p_k$. Define the Pearson statistic $X^2 = \sum_{i=1}^k \frac{(O_i - E_i)^2}{E_i}.$ Under the null, $X^2\to_d\chi^2_{k-1}$ as $n\to\infty$. The test rejects the null at level $\alpha$ when $X^2$ exceeds the $1-\alpha$ quantile of $\chi^2_{k-1}$.

The observed counts $(O_1,\dots,O_k)$ are multinomial under the null. The standardized cell residuals $(O_i - E_i)/\sqrt{E_i}$ are asymptotically Normal with covariance matrix $I - \sqrt p \sqrt p^\top$ (a projection onto the orthogonal complement of $\sqrt p = (\sqrt{p_1},\dots,\sqrt{p_k})$). The Pearson statistic is the squared Euclidean norm of this random vector, which is the squared length of a standard Normal projected onto a $(k-1)$-dimensional subspace, hence $\chi^2_{k-1}$.

This is the first asymptotic test in classical statistics and remains the most commonly used test for categorical data. Applications: testing whether a die is fair, whether observed nucleotide frequencies match a genome model, whether an empirical distribution matches a hypothesized one (after binning). The degrees-of-freedom rule is "number of cells minus one" because the cell counts are constrained to sum to $n$, removing one degree of freedom.

The asymptotic Chi-squared requires each expected count $E_i = np_i$ to be at least roughly 5; the small-sample rule of thumb is borrowed from Cochran. With cells of expected count below 5, the asymptotic approximation is poor and exact tests (Fisher's exact test, Monte-Carlo permutation) should replace it. Estimating cell probabilities from the data (rather than fixing them under the null) consumes additional degrees of freedom: the asymptotic distribution becomes $\chi^2_{k - 1 - r}$ where $r$ is the number of parameters estimated. This is the case in the next theorem.

For an $r\times c$ contingency table with observed counts $O_{ij}$, row totals $R_i$, and column totals $C_j$, let $E_{ij} = R_i C_j / n$ be the expected counts under independence. The Pearson statistic $X^2 = \sum_{i=1}^r\sum_{j=1}^c\frac{(O_{ij} - E_{ij})^2}{E_{ij}}$ satisfies $X^2\to_d\chi^2_{(r-1)(c-1)}$ under the null hypothesis of independence as $n\to\infty$. The test rejects at level $\alpha$ when $X^2$ exceeds the $1-\alpha$ quantile of $\chi^2_{(r-1)(c-1)}$.

Under independence, $p_{ij} = p_{i\cdot}p_{\cdot j}$. The MLE of the row and column marginals consumes $r - 1 + c - 1$ degrees of freedom (the marginals must sum to one, removing one constraint each). The total degrees of freedom are $rc - 1$ (for the multinomial constraint) minus $r - 1 - c + 1$ (for the estimated parameters), giving $rc - 1 - (r - 1) - (c - 1) = (r-1)(c-1)$.

This is the canonical test for association in cross-classified data: whether smoking status is independent of lung-cancer status, whether browser is independent of geography, whether treatment is independent of response. The test is asymptotic; for sparse tables with expected cell counts below 5, prefer Fisher's exact test (for $2\times 2$) or a permutation test (for larger). When the null is rejected, follow up with standardized residuals $(O_{ij} - E_{ij})/\sqrt{E_{ij}}$ to locate the cells driving the rejection.

The Chi-squared test treats rows and columns symmetrically. If the data are not i.i.d. (e.g., repeated measures on the same subject across columns), the asymptotic distribution is wrong and the test is invalid. For paired binary data on the same subjects, use McNemar's test. For repeated measures with more than two columns, use Cochran's Q or a mixed model.