Hypothesis Testing for ML

The null hypothesis $H_0$ is the default assumption you are trying to disprove. In ML model comparison:

$H_0: \mu_A = \mu_B$

where $\mu_A$ and $\mu_B$ are the true expected performances of models A and B. The null says there is no real difference between the models.

The alternative hypothesis $H_1$ is what you conclude if you reject $H_0$.

• Two-sided: $H_1: \mu_A \neq \mu_B$ (the models differ)
• One-sided: $H_1: \mu_A > \mu_B$ (model A is better)

In ML, two-sided tests are more conservative and generally preferred unless you have a strong prior reason to test only one direction.

The p-value is the probability of observing a test statistic at least as extreme as the one computed from your data, assuming $H_0$ is true:

$p = P(\text{test statistic} \geq t_{\text{obs}} \mid H_0)$

A small p-value means the observed data is unlikely under $H_0$, which is evidence against $H_0$.

The p-value is not the probability that $H_0$ is true. It is the probability of the data given $H_0$, not the probability of $H_0$ given the data.

The significance level $\alpha$ is the threshold for rejecting $H_0$. If $p \leq \alpha$, we reject $H_0$ and declare the result "statistically significant." The standard choice is $\alpha = 0.05$, meaning we accept a 5% chance of falsely rejecting a true null hypothesis.

A $(1-\alpha)$ confidence interval for a parameter $\theta$ is a random interval $[L, U]$ such that:

$P(\theta \in [L, U]) = 1 - \alpha$

For the difference in model performance $\delta = \mu_A - \mu_B$, a 95% CI provides a range of plausible values for the true difference. If the CI excludes zero, the difference is significant at $\alpha = 0.05$.

Confidence intervals are more informative than p-values alone because they communicate both statistical significance and effect size.

The False Discovery Rate is the expected proportion of false positives among all rejected hypotheses:

$\text{FDR} = \mathbb{E}\left[\frac{\text{false rejections}}{\text{total rejections}}\right]$

The Benjamini-Hochberg (BH) procedure controls FDR at level $q$:

1. Order the $m$ p-values: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$
2. Find the largest $k$ such that $p_{(k)} \leq \frac{k}{m} \cdot q$
3. Reject all hypotheses with $p_{(i)} \leq p_{(k)}$

BH is less conservative than Bonferroni and is preferred when testing many hypotheses (e.g., comparing models on 50 datasets).

Given $n$ test examples, let $d_i = \ell(A, x_i) - \ell(B, x_i)$ be the difference in loss between models A and B on example $i$. The paired t-test statistic is:

$t = \frac{\bar{d}}{s_d / \sqrt{n}}, \quad \bar{d} = \frac{1}{n}\sum_i d_i, \quad s_d^2 = \frac{1}{n-1}\sum_i (d_i - \bar{d})^2$

Under $H_0: \mathbb{E}[d_i] = 0$, the statistic $t$ follows a $t$-distribution with $n-1$ degrees of freedom (approximately, assuming $d_i$ are i.i.d.).

A non-parametric alternative to the paired t-test. It does not assume normality of the differences $d_i$. Instead, it ranks the absolute differences $|d_i|$ and compares the sum of ranks for positive vs. negative differences. Use this when the differences are not approximately normal (e.g., heavy-tailed or skewed distributions).

When the distribution of the test statistic is unknown or the sample is small:

1. Compute the observed test statistic $T_{\text{obs}}$ (e.g., difference in accuracy)
2. Generate $B$ bootstrap resamples under $H_0$ (e.g., by permuting model labels)
3. Compute $T_b^*$ for each resample
4. The p-value is the fraction of bootstrap statistics at least as extreme as the observed: $p = \frac{1}{B}\sum_{b=1}^B \mathbf{1}[|T_b^*| \geq |T_{\text{obs}}|]$

Bootstrap tests are flexible: they work for any test statistic and make minimal distributional assumptions.