Analysis of Variance

The total sum of squares decomposes as $\underbrace{\sum_{k=1}^K \sum_{j=1}^{n_k} (Y_{kj} - \bar Y)^2}_{\text{SS}_{\text{tot}}} \;=\; \underbrace{\sum_{k=1}^K n_k (\bar Y_k - \bar Y)^2}_{\text{SS}_{\text{between}}} \;+\; \underbrace{\sum_{k=1}^K \sum_{j=1}^{n_k} (Y_{kj} - \bar Y_k)^2}_{\text{SS}_{\text{within}}}.$ The decomposition is purely algebraic and does not require any distributional assumption.

The decomposition is the geometric heart of ANOVA: $\text{SS}_{\text{between}}$ is the squared distance from the group-mean fit to the grand-mean fit, and $\text{SS}_{\text{within}}$ is the squared distance from the data to the group-mean fit. Under iid normal data with equal variances, these two squared distances are independent and have known chi-squared distributions; their ratio gives the $F$ statistic.

The decomposition is exact. The downstream $F$-distribution claim is what fails under non-normal data or unequal variances; the decomposition itself holds for any partition of the data.

Under $H_0 : \mu_1 = \cdots = \mu_K = \mu$ and the iid $N(\mu, \sigma^2)$ assumption, $F = \frac{\text{SS}_{\text{between}} / (K - 1)}{\text{SS}_{\text{within}} / (N - K)} = \frac{\text{MS}_{\text{between}}}{\text{MS}_{\text{within}}} \;\sim\; F_{K - 1,\, N - K}.$ The two sums of squares satisfy $\frac{\text{SS}_{\text{between}}}{\sigma^2} \sim \chi^2_{K - 1}, \quad \frac{\text{SS}_{\text{within}}}{\sigma^2} \sim \chi^2_{N - K},$ and they are independent.

Project the data onto the subspace of vectors that are constant within each group (this gives the group means) and onto its orthogonal complement (this gives the within-group residuals). Under joint normality with equal variances, the two projections are jointly normal with zero cross-covariance, hence independent (see multivariate normal). The squared norms are independent chi-squared with degrees of freedom equal to the dimensions of the two subspaces.

This is one of the cleanest exact-distribution results in classical statistics. Under the stated assumptions, the test has level exactly $\alpha$ for any finite sample size $N$. The same construction underlies the $F$ tests in linear regression, ANCOVA, repeated-measures, and split-plot designs; they differ only in the choice of nested subspaces.

Three assumptions can break. (1) Non-normality. The $F$ test tolerates mild non-normality when the sample sizes are equal, but its level and power degrade when sample sizes are unequal. (2) Unequal variances (heteroscedasticity). The $F$ statistic no longer has an $F$ distribution; use the Welch correction below. (3) Non-independence (e.g., repeated measurements, clustered data). The decomposition itself is fine, but the chi-squared degrees of freedom are wrong; use a mixed model.

Define weights $w_k = n_k / S_k^2$ where $S_k^2$ is the within-group sample variance, $w_\cdot = \sum_k w_k$, and the weighted grand mean $\tilde Y = \sum_k w_k \bar Y_k / w_\cdot$. The Welch statistic is $F^* = \frac{\sum_k w_k (\bar Y_k - \tilde Y)^2 / (K - 1)}{1 + \frac{2(K - 2)}{K^2 - 1} \sum_k \frac{(1 - w_k / w_\cdot)^2}{n_k - 1}}.$ Under the equal-means null and approximate normality, $F^*$ is approximately $F_{K - 1, \nu^*}$ where $\nu^* = \frac{K^2 - 1}{3 \sum_k \frac{(1 - w_k / w_\cdot)^2}{n_k - 1}}.$

The naive equal-variance pooled estimate of $\sigma^2$ is replaced by a weighted average that gives more weight to groups with smaller variance. The degrees of freedom are then adjusted to match the first two moments of the resulting distribution.

Welch's correction is the default ANOVA in most statistical software (e.g., oneway.test in R) when variances are not assumed equal. For two groups, the same idea gives Welch's $t$-test, which is now the recommended replacement for the equal-variance two-sample $t$-test in nearly all applied contexts.

The Welch approximation is good when sample sizes are at least moderate (each $n_k \geq 5$ is a reasonable rule of thumb). For very small group sizes the approximation can be poor and a permutation or bootstrap alternative is safer.

With $N = I J n$ observations, the total sum of squares decomposes orthogonally into four components: $\text{SS}_{\text{tot}} \;=\; \text{SS}_A + \text{SS}_B + \text{SS}_{AB} + \text{SS}_E,$ where $\text{SS}_A = J n \sum_i (\bar Y_{i \cdot \cdot} - \bar Y)^2,\quad \text{SS}_B = I n \sum_j (\bar Y_{\cdot j \cdot} - \bar Y)^2,$ $\text{SS}_{AB} = n \sum_{i, j}(\bar Y_{ij \cdot} - \bar Y_{i \cdot \cdot} - \bar Y_{\cdot j \cdot} + \bar Y)^2,\quad \text{SS}_E = \sum_{i, j, k}(Y_{ijk} - \bar Y_{ij \cdot})^2.$ Under $H_0$ for each of the three effects, the corresponding $F$-statistic $F_{\text{effect}} = \frac{\text{SS}_{\text{effect}} / \text{df}_{\text{effect}}}{\text{SS}_E / (I J (n - 1))}$ has an exact $F$ distribution with degrees of freedom $(\text{df}_{\text{effect}}, I J (n - 1))$, where $\text{df}_A = I - 1$, $\text{df}_B = J - 1$, $\text{df}_{AB} = (I - 1)(J - 1)$.

The four components are orthogonal projections onto four nested subspaces. The dimension counts give the degrees of freedom. Under jointly normal data, the chi-squared distributions are independent (the multivariate normal independence property again) and the ratio is $F$.

This is the design-of-experiments core. The interaction sum of squares $\text{SS}_{AB}$ is the part of the variability that is not explained by the main effects acting additively; a significant interaction means the effect of one factor depends on the level of the other. Without checking the interaction, additive main-effects conclusions can be misleading.

Unbalanced designs (unequal $n$ per cell) break the orthogonality of the decomposition. Type I, II, and III sums of squares (sequential, hierarchical, partial) become different and the choice depends on the scientific question. Modern software defaults vary; specify the type explicitly in any reported analysis.