Lyapunov-Based Machine Learning for Chaos

Why This Matters

Standard MSE training on chaotic trajectory data fails for a structural reason. Pointwise prediction loss collapses past one Lyapunov time: any model that nails the next $0.1\,\tau_L$ accumulates exponential error after that, and the gradient signal becomes uninformative about the long-term geometry of the attractor. The model can be Pareto-optimal on short-horizon MSE while producing trajectories that drift off the attractor, fail energy conservation, or settle into spurious fixed points.

The fix is to train on geometric and dynamical invariants, not just trajectory samples. Match the Lyapunov spectrum. Enforce symplectic structure where the true system is Hamiltonian. Constrain the learned vector field to known conservation laws. The result is a model that may have worse one-step error but stays on the right manifold for arbitrary horizons.

This matters wherever ML meets physical dynamics: weather emulators, plasma simulators, molecular dynamics surrogates, climate downscaling, fusion control. The Lyapunov-time wall is real; the engineering question is what you ask the model to preserve once you cross it.

Core Ideas

Hamiltonian neural networks (HNN). Greydanus, Dzamba, and Yosinski (NeurIPS 2019; arXiv 1906.01563) parameterize a scalar $H_\theta(\mathbf{q}, \mathbf{p})$ and define the vector field by Hamilton's equations $\dot{\mathbf{q}} = \partial H/\partial \mathbf{p}$, $\dot{\mathbf{p}} = -\partial H/\partial \mathbf{q}$. Training matches observed time derivatives to those induced by $H_\theta$. Energy is conserved by construction up to integrator error, even off the training distribution. The cost is the architectural commitment: HNNs only apply where a Hamiltonian formulation exists.

Lagrangian neural networks (LNN). Cranmer et al. (ICLR 2020 workshop; arXiv 2003.04630) parameterize a Lagrangian $L_\theta(\mathbf{q}, \dot{\mathbf{q}})$ and recover dynamics through the Euler-Lagrange equations, which require inverting the mass matrix $\partial^2 L/\partial \dot{\mathbf{q}}^2$. LNNs handle systems with non-Cartesian coordinates and constraints that HNNs struggle with, at the cost of a Hessian inversion in the forward pass.

Lyapunov-spectrum matching. Add a loss term penalizing the difference between the maximal Lyapunov exponent of the learned model and the value estimated from data via the Wolf or Rosenstein algorithms. This is more expensive than trajectory MSE: each gradient step requires propagating a tangent vector or solving a variational equation along a sampled trajectory. The payoff is that the learned attractor has the right stretching rate, so short-horizon predictions plus invariant statistics both match.

Reservoir computing for chaos. Pathak, Hunt, Girvan, Lu, and Ott (Physical Review Letters 120, 2018; arXiv 1710.07313) showed that echo-state networks can predict chaotic flows several Lyapunov times into the future and, more importantly, reproduce the climatology and Lyapunov spectrum of the true system. This established the empirical baseline that more elaborate architectures are measured against. The result is sharp: prediction skill extends to roughly $5$ to $8\,\tau_L$ for Kuramoto-Sivashinsky, then collapses as expected.

The Lyapunov-time wall. Any deterministic model, regardless of capacity, loses meaningful pointwise predictive skill after a few Lyapunov times. Initial condition uncertainty $\delta(0)$ saturates the attractor diameter at time $t \approx \lambda_1^{-1} \log(D/\delta(0))$. Once you hit the wall, the correct objective shifts to ensemble forecasts, distributional metrics, and invariant statistics. Pretending the wall does not exist is the most common failure mode of ML weather and climate work.

Common Confusions

Watch Out

Energy conservation does not imply correct dynamics

A neural network that conserves energy exactly can still produce trajectories that miss the attractor or settle into the wrong invariant manifold. Symplectic structure is necessary but not sufficient. Validation needs Lyapunov spectra, power spectra, and invariant measure comparisons, not just energy drift plots.

Watch Out

Short-horizon MSE overstates skill on chaotic systems

A model with one-step MSE smaller than baseline can produce climatologies that look nothing like the true system. Short-horizon point error is the wrong metric past $\tau_L$. Report invariant statistics: marginal distributions, autocorrelation, Lyapunov exponents, and attractor reconstruction quality.

References

Related Topics

• Nonlinear Dynamics and Chaos Fundamentals
• Reservoir Computing and Echo State Networks
• Physics-Informed Neural Networks
• Neural ODEs
• Stochastic Differential Equations