Classical ODEs: Existence, Stability, and Numerical Methods

The IVP $\dot{x} = f(t, x), x(t_0) = x_0$ has a unique solution $x: [t_0, t_0 + \alpha] \to \mathbb{R}^d$ where $\alpha = \min(a, b/M)$.

Lipschitz continuity in $x$ controls how fast trajectories can separate. If two solutions started at the same point could diverge, then locally near that point $f$ would have to behave worse than linearly, contradicting the Lipschitz bound. The condition $\alpha = \min(a, b/M)$ guarantees the trajectory cannot leave the rectangle before time $\alpha$ elapses.

Picard-Lindelof is the gateway theorem. Without it, "the solution to this ODE" is an undefined phrase. Every scientific ML method that integrates a learned vector field $f_\theta$ implicitly assumes $f_\theta$ is Lipschitz, otherwise the trajectory it computes corresponds to nothing well-defined. Most neural networks with smooth activations (tanh, softplus, GELU) satisfy this; ReLU is Lipschitz but not differentiable at the origin, which is why ReLU-based neural ODEs work in practice but require care.

Lipschitz continuity is sufficient but not necessary. The classical counterexample is $\dot{x} = \sqrt{|x|}, x(0) = 0$, where $f$ is continuous but only Holder-1/2 continuous in $x$ near the origin. Two distinct solutions exist: $x \equiv 0$ and $x(t) = t^2/4$. Uniqueness fails precisely because the Lipschitz constant blows up at $x = 0$. Peano's theorem still gives existence under continuity alone, but uniqueness requires Lipschitz or similar regularity (Osgood's condition is the sharp generalization).

If $u(t) \leq \alpha(t) + \int_{t_0}^{t} \beta(s) u(s)\, ds$ for all $t \in [t_0, T]$, then:

$u(t) \leq \alpha(t) + \int_{t_0}^{t} \alpha(s) \beta(s) \exp\!\left(\int_s^t \beta(r)\, dr\right) ds.$

When $\alpha(t) \equiv \alpha_0$ is constant, this simplifies to $u(t) \leq \alpha_0 \exp\!\left(\int_{t_0}^{t} \beta(s)\, ds\right)$.

Gronwall converts an integral inequality into a closed-form pointwise bound. Apply it to the difference of two solutions of $\dot{x} = f(t,x)$ with Lipschitz constant $L$: $\|x_1(t) - x_2(t)\| \leq \|x_1(t_0) - x_2(t_0)\| e^{L(t - t_0)}$. Two trajectories starting close stay close, but only with an exponential-in-time bound. Over a long horizon with large $L$, "close" becomes meaningless quickly.

This bound governs every numerical-stability and adversarial-perturbation argument involving ODE flows. For neural ODEs, $L$ is roughly the operator norm of the Jacobian $\partial f_\theta/\partial h$. If this is large, gradient computations using the adjoint method accumulate error exponentially in $L \cdot T$. Lipschitz regularization of $f_\theta$ keeps $L$ small and keeps the system numerically tractable.

Let $\dot{V}(x) = \nabla V(x)^\top f(x)$ be the time derivative of $V$ along trajectories.

1. If $\dot{V}(x) \leq 0$ on $D$, then $x^* = 0$ is Lyapunov stable.
2. If $\dot{V}(x) < 0$ on $D \setminus \{0\}$, then $x^* = 0$ is asymptotically stable.

Treat $V$ as a generalized energy. If trajectories cannot increase $V$, they cannot escape the level sets $\{V \leq c\}$, so they remain near the equilibrium. If $V$ strictly decreases, every trajectory eventually reaches the bottom, which is the equilibrium.

Lyapunov is how stability is proven without solving the ODE. In control theory and reinforcement learning, the same machinery underlies stability of feedback policies. In ML training, treating $V(\theta) = \mathcal{L}(\theta) - \mathcal{L}^*$ and analyzing $\dot{V}$ under gradient flow is the standard convergence-proof template (see stability and optimization dynamics).

Constructing $V$ is an art. There is no algorithm for finding a Lyapunov function for a generic nonlinear system; the negative result of Lyapunov's method is uninformative because it might just mean you picked the wrong $V$. For LTI systems $\dot{x} = Ax$, $V(x) = x^\top P x$ with $P$ solving the Lyapunov equation $A^\top P + PA = -Q$ (any $Q \succ 0$) works iff all eigenvalues of $A$ have negative real part.