Early Stopping vs. Weight Decay

What Each Does

Both methods limit the effective capacity of a model to prevent overfitting. They operate through different mechanisms.

Early stopping monitors a validation metric during training and halts optimization when the metric stops improving (or begins degrading). The model parameters at the best validation checkpoint are used for evaluation. No modification to the loss function or optimizer is needed.

Weight decay adds a penalty proportional to the squared weight magnitude to the loss:

$\mathcal{L}_{\text{wd}}(\theta) = \mathcal{L}(\theta) + \frac{\lambda}{2} \|\theta\|_2^2$

The gradient update becomes $\theta_{t+1} = (1 - \lambda \eta) \theta_t - \eta \nabla \mathcal{L}(\theta_t)$, where $\eta$ is the learning rate. The factor $(1 - \lambda \eta)$ shrinks weights toward zero at every step. For adaptive optimizers like Adam, decoupled weight decay (AdamW) applies the shrinkage separately from the gradient scaling.

The Equivalence for Linear Models

For linear regression with gradient descent starting from $\theta_0 = 0$, early stopping after $T$ steps is equivalent to L2 regularization with $\lambda \propto 1/T$.

Consider the loss $\mathcal{L}(\theta) = \frac{1}{2}\|X\theta - y\|^2$ with gradient $\nabla \mathcal{L} = X^T(X\theta - y)$. In the eigenbasis of $X^TX$ with eigenvalues $\lambda_1, \ldots, \lambda_p$, the gradient descent update for eigencomponent $j$ is:

$\theta_j^{(T)} = \frac{1}{\lambda_j}(1 - (1 - \eta \lambda_j)^T) \hat{\theta}_j^{\text{OLS}}$

The factor $(1 - (1-\eta\lambda_j)^T)$ acts as a filter. For small $T$, components with small eigenvalues $\lambda_j$ are heavily attenuated, exactly like the L2 filter $\lambda_j / (\lambda_j + \alpha)$. Early stopping implicitly penalizes directions of low curvature, which are the directions that L2 regularization also penalizes.

This equivalence is exact in the limit of small learning rate and provides the theoretical foundation for understanding early stopping as implicit regularization.

Side-by-Side Comparison

When Each Wins

Early stopping wins: limited hyperparameter budget

Early stopping requires tuning only the patience parameter and the validation frequency. Weight decay requires choosing $\lambda$, which interacts with the learning rate, batch size, and model architecture. When compute is limited and you cannot afford a grid search over $\lambda$, early stopping provides regularization with minimal tuning.

Early stopping wins: compute savings

Early stopping literally trains less. If validation loss plateaus at epoch 50 out of 200 planned epochs, you save 75% of training compute. Weight decay runs for the full training schedule. For exploratory experiments where you do not know the right training duration, early stopping provides automatic duration selection.

Weight decay wins: no validation set needed

Early stopping requires a held-out validation set to monitor. Weight decay regularizes through the objective function and needs no validation data during training (though validation is still used for other hyperparameter decisions). In low-data regimes where every sample matters for training, weight decay avoids the data split.

Weight decay wins: deep networks with complex loss landscapes

The implicit L2 equivalence holds for linear models and breaks down for deep networks with non-convex loss surfaces. In deep networks, early stopping and weight decay have qualitatively different effects. Weight decay biases the solution toward a specific region of parameter space (small norm). Early stopping biases the solution toward the initialization region of parameter space. These are different inductive biases, and weight decay's explicit pressure toward small weights often provides better generalization in practice.

Both together: the standard practice

Modern deep learning uses both simultaneously. AdamW provides explicit weight decay, and training is monitored with validation loss for early stopping or learning rate scheduling. The two methods are complementary: weight decay controls where in parameter space the optimizer searches, and early stopping controls how long it searches.

The Linear Equivalence Breaks Down

For deep networks, early stopping is not equivalent to L2 regularization for several reasons:

1. Non-convexity: the loss landscape has multiple minima. Early stopping finds a different minimum than weight decay, not just a different point along the same trajectory.
2. Initialization dependence: early stopping keeps parameters near initialization. With standard initialization (e.g., Kaiming, Xavier), this means weights stay near their random initial scale. Weight decay pushes weights toward zero regardless of initialization.
3. Learning rate schedules: cosine decay, warmup, and other schedules change the effective regularization of early stopping in ways that have no L2 analog.
4. Adaptive optimizers: Adam's per-parameter learning rate scaling means different parameters are implicitly regularized differently by early stopping. Decoupled weight decay regularizes all parameters with the same $\lambda$.

Common Confusions

References

1. Prechelt, L. (1998). "Early Stopping - But When?" In Neural Networks: Tricks of the Trade, Springer. Lecture Notes in Computer Science, vol. 1524, pp. 55-69.
2. Yao, Y., Rosasco, L., and Caponnetto, A. (2007). "On Early Stopping in Gradient Descent Learning." Constructive Approximation, 26(2), 289-315. (Formal equivalence between early stopping and Tikhonov regularization.)
3. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. Section 7.8 (Early Stopping) and Section 7.1.1 (L2 Regularization).
4. Loshchilov, I. and Hutter, F. (2019). "Decoupled Weight Decay Regularization." ICLR 2019. (AdamW and the distinction between weight decay and L2 regularization.)
5. Ali, A., Kolter, J. Z., and Tibshirani, R. (2019). "A Continuous-Time View of Early Stopping for Least Squares Regression." AISTATS 2019. (Continuous-time analysis of the implicit regularization path.)
6. Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning, 2nd ed. Springer. Section 7.10 (Cross-validation and effective number of parameters).