Equilibrium and Implicit-Layer Models

A DEQ replaces the recurrence $h^{(l+1)} = f_\theta(h^{(l)})$ with the fixed-point equation:

$h^* = f_\theta(h^*)$

By the Banach fixed-point theorem, if $f_\theta$ is a contraction (i.e., $\|f_\theta(a) - f_\theta(b)\| \leq \gamma \|a - b\|$ for some $\gamma < 1$), then:

1. A unique fixed point $h^*$ exists.
2. Fixed-point iteration $h^{(l+1)} = f_\theta(h^{(l)})$ converges to $h^*$ from any initialization.
3. The convergence rate is $\|h^{(l)} - h^*\| \leq \gamma^l \|h^{(0)} - h^*\|$.

In practice, the fixed point is found using Anderson acceleration or Broyden's method rather than simple iteration.

Think of a standard ResNet as running a simulation forward in time. A DEQ says: I do not care about the trajectory, only the final equilibrium. If the system is stable (contractive), the equilibrium is unique regardless of starting point. This is like asking "what temperature will this room reach?" instead of simulating every second of heat transfer.

The equilibrium $h^*$ is the representation that an infinitely deep weight-tied network would produce. You compute it by solving an equation, not by running infinite layers.

DEQ models achieve competitive performance with explicit-depth transformers while using $O(1)$ memory in depth (only the fixed point and solver state are stored, not $L$ intermediate activations). This makes very deep effective computations feasible on limited hardware. The fixed-point perspective also connects neural networks to classical numerical analysis: the solver can use decades of research on efficient fixed-point methods.

The contraction assumption is critical. If $f_\theta$ is not contractive (Lipschitz constant $\geq 1$), the fixed-point iteration may diverge, oscillate, or converge to different points from different initializations. In practice, spectral normalization or careful initialization is used to ensure contractivity. Training can be unstable because the Lipschitz constant varies with $\theta$, and a parameter update can push the model from contractive to non-contractive.

Given a fixed point $h^* = f(h^*, \theta)$ and a loss $L(h^*)$, the gradient with respect to $\theta$ is:

$\frac{dL}{d\theta} = \frac{\partial L}{\partial h^*} \left(I - \frac{\partial f}{\partial h}\bigg|_{h^*}\right)^{-1} \frac{\partial f}{\partial \theta}\bigg|_{h^*}$

This is derived by differentiating $h^* = f(h^*, \theta)$ with respect to $\theta$:

$\frac{dh^*}{d\theta} = \frac{\partial f}{\partial h}\frac{dh^*}{d\theta} + \frac{\partial f}{\partial \theta}$

Solving: $\frac{dh^*}{d\theta} = \left(I - \frac{\partial f}{\partial h}\right)^{-1} \frac{\partial f}{\partial \theta}$.

The matrix inverse $(I - J_f)^{-1}$ (where $J_f = \partial f / \partial h$) is computed via a linear solve, not explicit inversion.

Standard backprop differentiates through each layer sequentially: the gradient flows backward through all $L$ layers. Implicit differentiation bypasses this entirely. It says: whatever path the forward computation took to reach $h^*$, the gradient depends only on the Jacobian of $f$ at the fixed point, not on the iteration history.

This is why DEQ models have $O(1)$ memory for backprop: you do not need to store intermediate iterates. You only need the fixed point $h^*$ and the ability to solve a linear system involving $(I - J_f)$.

This decouples the forward solve from the backward pass. You can use any solver (Anderson, Broyden, Newton) for the forward pass, and the backward pass is always the same linear system. This is more flexible than neural ODEs where the backward pass must use the adjoint ODE. It also means the gradient is exact (up to the linear solve tolerance), regardless of how many forward iterations were used.

The linear system $(I - J_f)v = b$ requires $\|J_f\| < 1$ for the Neumann series $(I - J_f)^{-1} = I + J_f + J_f^2 + \cdots$ to converge. This is guaranteed when $f$ is contractive. If the contraction constant is close to 1, the linear system is ill-conditioned and the gradient can be noisy. In practice, the linear solve is done with iterative methods (conjugate gradient, GMRES) with early termination, trading exact gradients for stable approximate ones.