Lesson 3 of 3 · About 5 minutes

Choose on validation, report on test

Each time you choose a model using a score, that score becomes part of the selection process. Keep a separate test set for the final evaluation.

Try it

Give each dataset one job. Decide which examples can fit the model, which can choose its settings, and which can support the final report.

Which data belongs in each step?

You have three separate datasets. Assign one to each action in a simple model-selection workflow, then check your choices.

Cross-validation rotates the validation role across the development data; it does not spend the final test set. Ordinary shuffled folds suit independent examples from a similar distribution. Time series need time-aware splits, and related observations may need to stay in the same group. Fit preprocessing inside each training fold.

How cross-validation fits in

theorem visual

Cross-validation rotates the holdout instead of spending one fixed test set

$Each fold takes one turn as validation data. The average validation loss estimates generalization, but it is still a model-selection signal, not a final test score.$

K-fold score

$CV_{K} = \frac{1}{n} \sum_{k = 1}^{K} \sum_{z_{i} \in S_{k}} ℓ (A (S ∖ S_{k}), z_{i})$

$Train on K - 1 folds, validate on the remaining fold, then average.$

Bias tradeoff

$n_{train} = n (1 - 1/ K)$

$Large K trains on more data, so the estimate is less pessimistic.$

Variance tradeoff

$K = n is LOO-CV$

$Leave-one-out has low bias but highly correlated fits, so model selection can still be noisy.$