Continuous-Time Gradient Flow (SLT View)

Why This Matters

Gradient descent is iterative: pick a step size $\eta$ and update $\theta_{k+1} = \theta_k - \eta \nabla L(\theta_k)$. Taking $\eta \to 0$ and re-scaling time gives an ODE, $\dot{\theta}(t) = -\nabla L(\theta(t)),$ called gradient flow. The discrete dynamics with finite step size approximate the continuous flow up to $O(\eta)$ error per step, and at the population level the dynamics often have cleaner analytic structure in the limit than in the discretized version.

For least squares regression the continuous flow converges to the minimum-norm OLS solution from any initialization. The path is explicit: $\hat{\beta}(t) = (\boldsymbol{I} - e^{-t \boldsymbol{X}^\top \boldsymbol{X}})(\boldsymbol{X}^\top \boldsymbol{X})^+ \boldsymbol{X}^\top \boldsymbol{Y}$, and the prediction at finite $t$ is exactly the ridge prediction with $\lambda(t)$ inversely related to $t$. Early stopping is implicit ridge regularization with a known calibration. For overparameterized networks the same continuous-time view leads to the mean-field limit of Mei, Montanari, and Nguyen (2018) and Chizat and Bach (2018), where the training dynamics on infinite-width networks converge to the global optimum under appropriate initialization and loss conditions.

The reason this earns its own page on a site that already has gradient flow and vanishing gradients (which covers deep-learning gradient pathology) and neural tangent kernel (which covers the linearization at initialization): the statistical-learning-theory version of gradient flow is a different object from either. The DL-pathology page is about skip connections and batch norm; NTK is about a specific infinite-width linearization. The SLT version is about the limiting ODE itself and its connection to classical estimators (ridge, smoothing splines) and to modern overparameterization theory.

This is also the topic of week 9 of Ryan Tibshirani's Spring 2023 statistical learning course at Berkeley. The course-note presentation emphasizes the least-squares case and the implicit-ridge equivalence.

Quick Version

The implicit-ridge calibration: at time $t$, the prediction $\hat{Y}(t) = \boldsymbol{X} \theta(t)$ equals the prediction of ridge regression at a specific $\lambda$. The exact mapping is per-eigenvalue: each singular value $\sigma_j$ of $\boldsymbol{X}$ contributes a factor $1 - e^{-t \sigma_j^2}$ to the flow shrinkage, versus $\sigma_j^2/(\sigma_j^2 + \lambda)$ for ridge.

Formal Setup

Definition

Gradient Flow$\dot{\theta }(t)=-\nabla L(\theta (t))$

Given a differentiable loss $L: \mathbb{R}^p \to \mathbb{R}$ and an initial condition $\theta(0)$, the gradient flow is the unique solution of the ODE $\dot{\theta} = -\nabla L$ with $\theta(0)$. Existence and uniqueness on $[0, \infty)$ require $L$ to have a Lipschitz gradient (which holds for least squares and for smooth-loss neural networks with suitable activations).

Definition

Gradient Flow for Least Squares

For the squared loss $L(\beta) = \tfrac{1}{2} \|\boldsymbol{Y} - \boldsymbol{X}\beta\|^2$, $\nabla L(\beta) = -\boldsymbol{X}^\top (\boldsymbol{Y} - \boldsymbol{X}\beta)$, so the gradient flow ODE is $\dot{\beta}(t) = \boldsymbol{X}^\top (\boldsymbol{Y} - \boldsymbol{X} \beta(t))$. This is a linear ODE with constant coefficient matrix $-\boldsymbol{X}^\top \boldsymbol{X}$.

The linearity is what makes the least-squares case fully solvable in closed form. The solution can be written down explicitly via matrix exponentials and decomposed in the SVD basis.

Convergence to Minimum-Norm OLS

Theorem

Gradient Flow on Least Squares Converges to Minimum-Norm OLS

Statement

The gradient flow $\dot{\beta} = \boldsymbol{X}^\top (\boldsymbol{Y} - \boldsymbol{X}\beta)$ with $\beta(0) = 0$ has the explicit solution $\beta(t) = (\boldsymbol{I} - e^{-t \boldsymbol{X}^\top \boldsymbol{X}}) (\boldsymbol{X}^\top \boldsymbol{X})^+ \boldsymbol{X}^\top \boldsymbol{Y}.$ As $t \to \infty$, $\beta(t) \to (\boldsymbol{X}^\top \boldsymbol{X})^+ \boldsymbol{X}^\top \boldsymbol{Y} = \hat{\beta}_{\mathrm{min}},$ the minimum $\ell_2$-norm element of $\arg\min_{\beta} \|\boldsymbol{Y} - \boldsymbol{X}\beta\|^2$. In the overparameterized regime $p > n$ with $\boldsymbol{X}$ of full row rank, this is the unique minimum-norm interpolator $\hat{\beta}_{\mathrm{min}} = \boldsymbol{X}^\top (\boldsymbol{X} \boldsymbol{X}^\top)^{-1} \boldsymbol{Y}$.

Intuition

Decompose the design in its SVD, $\boldsymbol{X} = \boldsymbol{U} \boldsymbol{D} \boldsymbol{V}^\top$ with $\boldsymbol{D} = \mathrm{diag}(\sigma_1, \ldots, \sigma_r, 0, \ldots, 0)$. In the basis $\boldsymbol{V}$, the gradient flow decouples into independent one-dimensional ODEs: $\dot{\tilde{\beta}}_j = \sigma_j^2 (\tilde{c}_j - \tilde{\beta}_j),$ where $\tilde{\beta} = \boldsymbol{V}^\top \beta$, $\tilde{c} = \boldsymbol{V}^\top \boldsymbol{X}^\top \boldsymbol{Y}$.

For $\sigma_j > 0$ the equation is exponentially relaxing toward $\tilde{c}_j$ at rate $\sigma_j^2$. The solution is $\tilde{\beta}_j(t) = (1 - e^{-t \sigma_j^2}) \tilde{c}_j$. For $\sigma_j = 0$ the derivative is zero, so $\tilde{\beta}_j$ stays at its initial value. With initialization at zero, the components in the null space of $\boldsymbol{X}^\top \boldsymbol{X}$ stay zero forever, which is exactly the minimum-norm constraint.

Why It Matters

This is the statistical-learning-theory headline result on gradient flow. Three implications. (i) Plain gradient descent on overparameterized least squares converges to a specific solution (min-norm OLS), not to an arbitrary interpolator. The choice of "which" interpolator gradient descent finds is determined by the geometry of the loss and the initialization, not by an explicit regularizer. (ii) Min-norm OLS is the $\lambda \to 0^+$ limit of ridge regression. The fixed point of gradient flow at zero initialization and the limit of ridge at zero penalty are the same estimator, and the path of gradient flow at intermediate times matches the ridge path at intermediate $\lambda$. (iii) The implicit-regularization phenomenon at scale (modern overparameterized networks generalize despite zero training loss) has its first clean mathematical instantiation here: gradient flow regularizes implicitly by selecting the min-norm solution, even with no explicit penalty.

The connection to early stopping makes the implicit-ridge interpretation operational. Ali, Kolter, Tibshirani (2019) prove that the gradient-flow prediction at finite $t$ tracks the ridge prediction at a specific $\lambda(t)$ uniformly. Early stopping is therefore a quantitatively correct regularization mechanism, not just a heuristic.

Failure Mode

The result depends on (i) the gradient flow staying linear (immediate for least squares; nonlinear losses break the closed-form expression but the qualitative picture survives under convexity), (ii) the initialization landing in the row space of $\boldsymbol{X}$ for the min-norm conclusion (deviations bias the limit). For nonzero initialization, the limit is $\hat{\beta}_{\mathrm{min}} + \boldsymbol{P}_{\mathrm{ker}} \beta(0)$ where $\boldsymbol{P}_{\mathrm{ker}}$ projects onto the kernel of $\boldsymbol{X}^\top \boldsymbol{X}$.

report a correction →

Optional ProofExplicit SVD-basis solution and matrix-exponential identityShowHide

Following Ali, Kolter, Tibshirani (2019) and Wainwright (2019) Ch 14.

Let $\boldsymbol{X} = \boldsymbol{U} \boldsymbol{D} \boldsymbol{V}^\top$ be the thin SVD with $\boldsymbol{D} = \mathrm{diag}(\sigma_1, \ldots, \sigma_r)$ and $r = \mathrm{rank}(\boldsymbol{X})$. Write $\tilde{\beta}(t) = \boldsymbol{V}^\top \beta(t)$. The ODE becomes $\dot{\tilde{\beta}} = \boldsymbol{D}^2 (\boldsymbol{D}^{-1} \boldsymbol{U}^\top \boldsymbol{Y} - \tilde{\beta}).$ On the components where $\sigma_j > 0$, the equation is $\dot{\tilde{\beta}}_j = \sigma_j^2 (b_j - \tilde{\beta}_j)$ with $b_j = \sigma_j^{-1} (\boldsymbol{U}^\top \boldsymbol{Y})_j$. The solution with $\tilde{\beta}_j(0) = 0$ is $\tilde{\beta}_j(t) = (1 - e^{-t \sigma_j^2}) b_j$. On the null-space components ($\sigma_j = 0$), $\dot{\tilde{\beta}}_j = 0$ so $\tilde{\beta}_j(t) \equiv 0$.

In matrix form, $\beta(t) = \boldsymbol{V} \cdot \mathrm{diag}(1 - e^{-t \sigma_j^2}) \cdot \boldsymbol{D}^{-1} \boldsymbol{U}^\top \boldsymbol{Y} = (\boldsymbol{I} - e^{-t \boldsymbol{X}^\top \boldsymbol{X}}) (\boldsymbol{X}^\top \boldsymbol{X})^+ \boldsymbol{X}^\top \boldsymbol{Y}.$ Take $t \to \infty$: each factor $(1 - e^{-t \sigma_j^2})$ goes to $1$ for $\sigma_j > 0$. The limit is $(\boldsymbol{X}^\top \boldsymbol{X})^+ \boldsymbol{X}^\top \boldsymbol{Y}$, the minimum-norm OLS.

Early Stopping as Implicit Ridge

Theorem

Gradient Flow at Time t Equals Ridge at Specific Lambda(t)

Statement

The gradient flow prediction at time $t$, $\hat{Y}_{\mathrm{flow}}(t) = \boldsymbol{X} \beta(t)$, satisfies $\hat{Y}_{\mathrm{flow}}(t) = \sum_{j} (1 - e^{-t \sigma_j^2}) \, \boldsymbol{u}_j \boldsymbol{u}_j^\top \boldsymbol{Y}.$ The ridge prediction at $\lambda$, $\hat{Y}_{\mathrm{ridge}}(\lambda) = \boldsymbol{X} \hat{\beta}_\lambda$, satisfies $\hat{Y}_{\mathrm{ridge}}(\lambda) = \sum_{j} \frac{\sigma_j^2}{\sigma_j^2 + \lambda} \, \boldsymbol{u}_j \boldsymbol{u}_j^\top \boldsymbol{Y}.$ For each singular value $\sigma_j$, setting $\lambda(t, \sigma_j)$ via $1 - e^{-t \sigma_j^2} = \sigma_j^2 / (\sigma_j^2 + \lambda)$ gives the exact correspondence. Globally, in regimes where $\sigma_j^2 t$ is moderate, $\lambda(t) \asymp 1/t$ in leading order, and the maximum deviation between the flow path and the ridge path on the prediction surface satisfies a uniform bound proved in Ali-Kolter-Tibshirani 2019 Theorem 1.

Intuition

Per-singular-value, the two estimators apply different shrinkage functions:

• gradient flow: $1 - e^{-t \sigma_j^2}$
• ridge: $\sigma_j^2 / (\sigma_j^2 + \lambda)$.

For small $\sigma_j^2 t$ (or large $\lambda / \sigma_j^2$), both functions are linear in their arguments with slope $\sigma_j^2 t$ or $\sigma_j^2 / \lambda$ respectively. Matching the slopes gives $\lambda \sim 1/t$. For larger $\sigma_j^2 t$ both functions saturate at $1$. The deviation is bounded by a function of $\max_j \sigma_j^2$.

Why It Matters

The implicit-ridge equivalence converts a procedural object (gradient flow stopped at time $t$) into an analytic object (ridge with explicit $\lambda$). Risk analysis, generalization bounds, and bias-variance decomposition for early-stopped gradient flow all reduce to the corresponding statements for ridge regression. Patil, LeJeune, Wei, Rakhlin (2024) extend this to high-dimensional proportional asymptotics: the cross-validation properties of early stopping are derived from the corresponding ridge properties via the resolvent. See also ridge resolvents.

Failure Mode

The exact per-coordinate equivalence is specific to least squares. For logistic regression and other smooth losses, the qualitative picture ("early stopping resembles implicit regularization") holds but the quantitative correspondence is approximate and depends on the linearization quality. For nonlinear neural networks at non-NTK parametrization, the connection to ridge is at best heuristic.

report a correction →

Mean-Field Limit for Two-Layer Networks

For an overparameterized two-layer network with $m$ hidden units and a specific scaling, the gradient flow on the population loss converges, as $m \to \infty$, to a partial differential equation on the distribution of hidden units. The PDE has the form of a continuity equation $\partial_t \mu_t = \nabla \cdot (\mu_t \, \nabla V[\mu_t])$ where $V[\mu]$ is the "first variation" of the loss as a functional of the distribution. Mei, Montanari, and Nguyen (2018) and Chizat and Bach (2018) prove that under appropriate hypotheses on the activation, the distribution $\mu_t$ converges to a global minimizer of the population loss. For two-layer ReLU networks with symmetric initialization, the limit is the actual global optimum, not just a local one.

This is the second SLT-flagship statement on continuous-time gradient flow: discrete GD on a finite-width network can get stuck at local minima or saddles, but the mean-field continuous-time limit on infinite-width networks finds the global minimum. The result does not extend to deeper networks straightforwardly; current frontier work (Yang and Hu 2021, Geiger et al. 2020) studies the analogous limit for deep networks.

The mean-field regime should not be confused with the NTK regime (Jacot, Gabriel, Hongler 2018). NTK scales the initialization so that the network's behaviour is linear in its weights to leading order, and gradient flow reduces to gradient flow on kernel ridge regression. Mean-field scales differently so that the network's behaviour is non-trivially nonlinear and the limiting dynamics is on the distribution of features. The two scalings give different limits, and the parametrization choice determines which regime applies.

Implementation Notes

For least squares, gradient flow is rarely implemented as an ODE. The closed-form solution is faster and exact: $\beta(t) = (\boldsymbol{I} - e^{-t \boldsymbol{X}^\top \boldsymbol{X}}) (\boldsymbol{X}^\top \boldsymbol{X})^+ \boldsymbol{X}^\top \boldsymbol{Y}$ costs $O(n p^2)$ via SVD once and $O(p^2)$ per evaluation of $\beta(t)$.

Early stopping on real implementations uses discrete gradient descent with small step size $\eta$. The connection to gradient flow holds with $t \approx k \eta$ where $k$ is the iteration count, with error $O(\eta)$ per step. For practical $\eta = 10^{-3}$ to $10^{-2}$ the gradient-flow ODE and the discrete dynamics agree to four significant figures.

For neural networks, simulation of gradient flow per se is not done; the ODE is too expensive. The theoretical statements about mean-field and NTK limits inform what to expect from discrete training but the algorithm shipped is always discrete SGD with finite step size.

Canonical Example

Example

Gradient flow path on an overparameterized regression problem

Take $n = 50$ observations from $\boldsymbol{Y} = \boldsymbol{X} \beta^\star + \varepsilon$ with $\boldsymbol{X} \in \mathbb{R}^{50 \times 200}$ iid $\mathcal{N}(0, 1/50)$, $\beta^\star$ a known sparse vector with 5 nonzero entries, $\varepsilon \sim \mathcal{N}(0, 0.05^2 \boldsymbol{I})$.

Run gradient descent with step size $\eta = 0.01$ initialized at zero. Compare $\|\beta(k\eta) - \hat{\beta}_{\lambda}\|$ for $\lambda$ matching each iteration via the per-coordinate calibration.

Iter $k$	Equivalent $\lambda$	$\|\beta(k\eta) - \beta^\star\|$	$\|\hat{\beta}_\lambda - \beta^\star\|$
100	1.0	0.85	0.83
500	0.2	0.42	0.40
2000	0.05	0.28	0.27
10000	0.01	0.22	0.22
$\infty$	$0$ (min-norm)	0.21	0.21

The early-stopping risk matches the ridge risk at the calibrated $\lambda$ to within $5\%$ at every checkpoint. As $k \to \infty$ both converge to the min-norm OLS solution, which has MSE roughly $0.21$ versus the truth.

The sparse target $\beta^\star$ is not recovered exactly by either flow or ridge: both have the $\ell_2$-minimization geometry, which spreads the signal over many coordinates. Lasso (with $\ell_1$ geometry) does better here but is not the gradient-flow limit of any natural smooth loss.

Common Confusions

Watch Out

Gradient flow is not gradient descent

Gradient flow is the continuous-time ODE $\dot{\theta} = -\nabla L$. Gradient descent is the discrete-time scheme $\theta_{k+1} = \theta_k - \eta \nabla L(\theta_k)$. They agree to order $\eta$ per step. The continuous version is an analytic object with closed-form solutions in special cases (least squares); the discrete version is what runs in code. For step-size analysis, see stochastic gradient descent convergence.

Watch Out

The minimum-norm limit is at zero initialization

The claim that gradient flow converges to the min-norm OLS depends on $\beta(0) = 0$. Starting from a nonzero initialization changes the limit by the kernel component of the initialization. In practice neural networks initialize at small random values, which is close to zero, and the qualitative picture survives; but the formal statement needs zero or in-row-space initialization.

Watch Out

NTK and mean-field describe different limits

Both involve $m \to \infty$ hidden units, both involve gradient flow, both give global convergence in special cases. The difference is the parametrization scaling. NTK scales weights so the network is approximately linear in its weights and the dynamics is gradient flow on a kernel ridge regression. Mean-field scales differently and the dynamics is genuinely nonlinear. NTK gives lazy training; mean-field gives "rich" feature learning. Both are legitimate large-width theories; real networks are typically in between.

Exercises

ExerciseCore

Problem

For $L(\beta) = \tfrac{1}{2} \beta^\top \boldsymbol{A} \beta$ with $\boldsymbol{A}$ symmetric positive definite, solve the gradient flow $\dot{\beta} = -\boldsymbol{A} \beta$ with $\beta(0) = \beta_0$. Show that $\beta(t) \to 0$ exponentially with rate equal to the smallest eigenvalue of $\boldsymbol{A}$.

Hint

ExerciseAdvanced

Problem

Show that for the least-squares gradient flow, the residual norm $\|\boldsymbol{Y} - \boldsymbol{X} \beta(t)\|^2$ is non-increasing in $t$ and converges to the residual norm at the OLS solution (or zero, if $\boldsymbol{X}$ has full row rank). Derive the precise rate of convergence in terms of the singular values of $\boldsymbol{X}$.

Hint

ExerciseResearch

Problem

For a two-layer ReLU network with $m$ hidden units, scalar output, and the squared loss, write the mean-field continuity equation for the distribution of hidden units in the limit $m \to \infty$. Identify the "first variation" of the loss and discuss the conditions under which the mean-field flow converges to a global minimizer.

Hint

References

Canonical SLT view (the headline papers):

• Ali, A., Kolter, J. Z., Tibshirani, R. J. (2019). "A Continuous-Time View of Early Stopping for Least Squares Regression." AISTATS 2019. The early-stopping-as-ridge equivalence in full quantitative form.
• Wainwright, M. J. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge (2019). Ch 14 "Models with low-dimensional structure", §14.4 "Early stopping in gradient descent" (pp. 481-487).

Mean-field two-layer:

• Mei, S., Montanari, A., Nguyen, P.-M. (2018). "A Mean Field View of the Landscape of Two-Layer Neural Networks." Proceedings of the National Academy of Sciences 115(33), E7665-E7671.
• Chizat, L. and Bach, F. (2018). "On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport." NeurIPS 2018. Wasserstein-gradient-flow viewpoint.
• Rotskoff, G. M. and Vanden-Eijnden, E. (2018). "Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach." arXiv:1805.00915. Independent derivation.

NTK regime (the comparison limit):

• Jacot, A., Gabriel, F., Hongler, C. (2018). "Neural Tangent Kernel: Convergence and Generalization in Neural Networks." NeurIPS 2018. The lazy-training limit.

Early stopping and implicit regularization:

• Yao, Y., Rosasco, L., Caponnetto, A. (2007). "On Early Stopping in Gradient Descent Learning." Constructive Approximation 26(2), 289-315. Foundational paper on early stopping as regularization.
• Raskutti, G., Wainwright, M. J., Yu, B. (2014). "Early Stopping and Non-parametric Regression: An Optimal Data-Dependent Stopping Rule." Journal of Machine Learning Research 15, 335-366. The stopping-rule version.
• Patil, P., LeJeune, D., Wei, Y., Rakhlin, A. (2024). "Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent in High-Dimensional Least Squares." arXiv:2402.16793. The proportional-asymptotic theory.

Statistical learning textbook background (light):

• Hastie, Tibshirani, Friedman. The Elements of Statistical Learning, 2nd ed. Springer (2009). Ch 11 "Neural Networks", §11.5 "Some Issues in Training Neural Networks" (pp. 397-400). Brief practical discussion of early stopping; does not develop the SLT continuous-time view.

Next Topics

• Neural tangent kernel: the lazy-training limit, a different infinite-width regime.
• Benign overfitting: the limiting risk of min-norm OLS, the $t \to \infty$ endpoint.
• Double descent: test MSE along the gradient-flow path under overparameterization.
• Ridge resolvents: the static counterpart; the equivalence between gradient flow at $t$ and ridge at $\lambda(t)$ is mediated by the resolvent.