Autoencoders for Low-Dimensional Dynamical Structures

Why This Matters

Most high-dimensional dynamical systems live on a much lower-dimensional manifold. Fluid flows past obstacles have effective state dimension in the tens even when the simulation grid has $10^6$ cells. Neural population activity in motor cortex is well-described by a few latent factors. Climate fields evolve on quasi-geostrophic manifolds. The practical question is how to find that manifold and how to model the dynamics on it.

Three families of methods compete here, each with a different objective. Proper orthogonal decomposition (POD) and principal component analysis (PCA) find the linear subspace that maximizes captured variance. Dynamic mode decomposition (DMD, Schmid 2010) finds linear modes that best describe the temporal evolution under a best-fit linear operator. Autoencoders, including their Koopman-aware variants, find nonlinear coordinates in which the dynamics become more tractable, often approximately linear.

For ML, the relevance is twofold: low-dimensional embeddings make downstream control, prediction, and uncertainty quantification cheap, and the structure of the embedding tells you something about the underlying physics that the raw state hides.

Core Ideas

Dynamic mode decomposition. Schmid (Journal of Fluid Mechanics 656, 2010) introduced DMD as a linear-algebra procedure: given snapshot pairs $(\mathbf{x}_t, \mathbf{x}_{t+1})$, compute the best-fit linear operator $A$ satisfying $\mathbf{x}_{t+1} \approx A \mathbf{x}_t$ via least squares, then take its eigendecomposition. Each eigenpair $(\lambda_j, \boldsymbol{\phi}_j)$ gives a spatial mode $\boldsymbol{\phi}_j$ with a temporal frequency $\arg(\lambda_j)$ and growth rate $\log|\lambda_j|$. DMD is data-driven, fast, and extracts coherent structures that POD can miss when variance and dynamics disagree.

Koopman operator interpretation. The Koopman operator $\mathcal{K}$ acts on observables $g(\mathbf{x})$ via $(\mathcal{K} g)(\mathbf{x}) = g(F(\mathbf{x}))$, where $F$ is the (nonlinear) flow map. $\mathcal{K}$ is linear but infinite-dimensional. DMD is, in this view, a finite-dimensional approximation to $\mathcal{K}$ in a basis of linear observables. Extended DMD (Williams, Kevrekidis, Rowley 2015; arXiv 1408.4408) generalizes to user-chosen nonlinear dictionaries.

Deep Koopman / Deep DMD. Lusch, Kutz, and Brunton (Nature Communications 9, 2018; arXiv 1712.09707) train an autoencoder $\boldsymbol{\psi}: \mathbf{x} \mapsto \mathbf{z}$ together with a linear operator $K$ acting on latents, with a loss that enforces $\boldsymbol{\psi}(F(\mathbf{x})) \approx K \boldsymbol{\psi}(\mathbf{x})$, reconstruction $\boldsymbol{\psi}^{-1}(\boldsymbol{\psi}(\mathbf{x})) \approx \mathbf{x}$, and prediction over multiple steps. The encoder learns a Koopman-invariant subspace; the latent dynamics are linear by construction; long-horizon prediction reduces to repeated multiplication by $K$.

beta-VAE for disentangled dynamical modes. Higgins et al. (ICLR 2017) augment the VAE objective with a coefficient $\beta > 1$ on the KL term, encouraging factorized latent representations. Applied to dynamical data, this sometimes recovers latents that align with physical degrees of freedom (rotation angle, oscillation phase), although the disentanglement claim is fragile and identifiability is provably impossible without inductive biases (Locatello et al., ICML 2019). In dynamics, the better-grounded approach is to combine the VAE with a Koopman or neural-ODE prior on the latents.

POD and PCA contrast. POD (equivalent to PCA on snapshot data) is optimal for variance capture under a linear projection; it is energy-optimal but not dynamics-optimal. A high-energy POD mode can be slowly varying noise; a low-energy DMD mode can carry a critical instability. For control and prediction, DMD or Koopman embeddings dominate. For compression of static ensembles, POD is fine and faster.

Common Confusions

Watch Out

DMD is not just PCA on time-derivative data

Both extract spatial modes from snapshot matrices, but the objectives differ. PCA maximizes captured variance over the snapshot ensemble. DMD finds modes that diagonalize a best-fit linear evolution operator. They coincide only for purely periodic data with orthogonal modes. On quasi-periodic, transient, or nonnormal flows, DMD and PCA give different mode rankings; DMD is usually the right choice when you care about prediction or stability.

References

Related Topics

• Autoencoders
• Variational Autoencoders
• Principal Component Analysis
• Reservoir Computing and Echo State Networks
• Neural ODEs