Equivariant Deep Learning

A function $f: \mathcal{X} \to \mathcal{Y}$ is equivariant with respect to group $G$ if:

$f(\rho_X(g, x)) = \rho_Y(g, f(x)) \quad \forall g \in G, \; x \in \mathcal{X}$

Transforming the input, then applying $f$, gives the same result as applying $f$, then transforming the output. The function "commutes" with the group action.

Invariance is the special case where $\rho_Y$ is trivial: $f(\rho_X(g, x)) = f(x)$ for all $g$. The output does not change at all.

An equivariant function preserves the structure of transformations. If you rotate a molecule 90 degrees and then predict its energy, you should get the same energy as if you first predict and then (conceptually) rotate. If you rotate it and predict its dipole moment, the dipole should rotate by the same 90 degrees.

Invariance (energy does not change under rotation) and equivariance (dipole rotates with the molecule) are both useful, and which one you want depends on what you are predicting.

Equivariance is a hard constraint, not a soft regularizer. A network that is equivariant by construction will respect the symmetry perfectly on all inputs, not just approximately on training data. This is a strict generalization guarantee: the network cannot learn to violate the symmetry, even with adversarial data. This is why equivariant networks need dramatically less data than unconstrained networks for tasks with known symmetries.

The symmetry must be exact. If your data has approximate symmetry (e.g., images are roughly but not exactly rotation-invariant because of gravity), enforcing exact equivariance can hurt. The network cannot learn that "up" and "down" are different if you force rotational invariance. In such cases, data augmentation (soft symmetry) may outperform equivariant architectures (hard symmetry).

A linear map $W: \mathbb{R}^{d_{\text{in}}} \to \mathbb{R}^{d_{\text{out}}}$ that is equivariant with respect to representations $\rho_{\text{in}}$ and $\rho_{\text{out}}$ of $G$ satisfies:

$W \rho_{\text{in}}(g) = \rho_{\text{out}}(g) W \quad \forall g \in G$

The set of matrices satisfying this constraint is a linear subspace of $\mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$. The dimension of this subspace (the number of free parameters) is at most $d_{\text{in}} d_{\text{out}} / |G|$ for a finite group $G$.

The equivariance constraint forces parameter sharing. In a CNN, translation equivariance forces the same filter weights at every position, reducing parameters from $O(\text{image size} \times \text{filter size})$ to $O(\text{filter size})$. For rotation equivariance, the constraint forces the filter to be "steerable" (a linear combination of a fixed set of basis filters), further reducing parameters.

Fewer free parameters means the function class is smaller, which improves generalization via the bias-variance tradeoff. The bias is increased (you cannot represent symmetry-breaking functions), but the variance decreases (less overfitting) by exactly the right amount when the symmetry holds.

This is why equivariant networks work with less data: the parameter sharing from equivariance is not arbitrary compression, it is compression that perfectly matches the data symmetry. The number of parameters scales inversely with the group size, so larger symmetry groups give more compression. A $|G| = 8$ rotation group (discrete rotations of a square) gives $8\times$ fewer parameters. A continuous rotation group gives infinite compression (parameters depend only on the radial profile, not the angle).

Computing the equivariant subspace requires solving the intertwiner condition $W\rho_{\text{in}}(g) = \rho_{\text{out}}(g)W$ for all $g$, which requires knowledge of the group representations. For simple groups (translations, rotations, permutations), the representations are well-known. For complex or non-standard symmetries, finding the representations is a research problem in itself.