Grokking

On structured algorithmic tasks (modular arithmetic, permutation composition, polynomial evaluation), neural networks trained with SGD exhibit the following training dynamics:

1. Phase 1 (memorization): Training accuracy reaches 100% typically within $\sim 10^3$ to $\sim 10^4$ steps in the Power et al. (2022) modular arithmetic settings. Validation accuracy remains near chance.

2. Phase 2 (plateau): Training loss stays near zero. Validation accuracy stays near chance. This phase lasts $\sim 10^4$ to $10^6$ steps, strongly dependent on train fraction, weight decay, and optimizer.

3. Phase 3 (generalization): Validation accuracy rises sharply to near 100% over a relatively short window.

The transition from Phase 2 to Phase 3 is a phase transition: the model's internal representation undergoes a qualitative structural change.

The network first finds a memorization solution that stores training pairs in its weights. This is not a literal lookup table, but a distributed representation that achieves the same input-output mapping via high-dimensional geometry. This solution has high weight norm because it stores $n$ unrelated mappings. With weight decay applying constant pressure toward smaller weights, the network is slowly pushed toward lower-norm solutions. Eventually, it crosses a threshold where a compact, generalizing algorithm (e.g., the modular arithmetic circuit) becomes lower-loss than the memorization solution. The transition is sudden because the two solution types are structurally different. There is no smooth interpolation between a distributed memorization and a compact algorithm.

Grokking shows that optimization time can change what a network knows, not just how well it fits. This has implications for: training schedule design (early stopping may prevent generalization), regularization theory (weight decay is not just preventing overfitting, it is actively shaping the solution landscape), and mechanistic interpretability (you can study how networks transition from memorized to algorithmic representations).

Grokking is easiest to demonstrate on small, highly structured tasks (modular arithmetic with $\leq 100$ elements). It is unclear how much this phenomenon extends to large-scale, naturalistic training. Some evidence suggests analogous phenomena in larger models, but the clean phase-transition signature becomes less sharp. Do not assume all training runs will exhibit grokking if you just train long enough.

In the Power et al. (2022) setting, adding $\ell_2$ weight decay is a sufficient mechanism to produce grokking. With weight decay, the effective loss is:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{data}} + \frac{\lambda}{2} \|w\|^2$

After the memorization phase ($\mathcal{L}_{\text{data}} \approx 0$), the dominant gradient signal comes from the weight decay term, which pushes the weights toward the origin. This continuous pressure compresses the representation until the network transitions to a lower-norm solution that generalizes.

Empirical observation (not a theorem): Power et al. (2022) report that the grokking time $T_{\text{grok}}$ scales roughly as $T_{\text{grok}} \propto 1/\lambda$ for small $\lambda$ in this setting. This is empirical; no closed-form derivation exists. Alternative framings make the scaling a consequence of other quantities: Liu et al. (2023, "Omnigrok") cast grokking as governed by the ratio of weight norm to a critical value; Varma et al. (2023) cast it as a circuit efficiency ratio between memorizing and generalizing circuits.

Weight decay is sufficient, not necessary. Later work shows grokking can arise from other sources of compression or from a lazy-to-rich transition without explicit $\ell_2$ weight decay. See the FailureMode and the "When Grokking Happens" table for scope.

Weight decay is doing implicit architecture search. After memorization, the loss landscape near the memorizing solution looks flat (training loss is zero regardless of small weight changes). But the weight decay gradient keeps pulling the weights inward. Over time, this compression forces the network out of the memorization basin and into a generalization basin. Stronger weight decay means faster compression and faster grokking. Too much weight decay prevents memorization in the first place.

The deeper story is that some implicit or explicit force toward lower-complexity representations is what drives the transition. $\ell_2$ weight decay is the cleanest and most studied version, but it is not the only one.

This connects grokking to the broader theory of implicit bias. Gradient descent has specific biases: max-margin for logistic loss on separable data (Soudry et al. 2018), min-norm for squared loss. On algorithmic tasks with weight decay (Power 2022, Nanda 2023), the min-norm interpolator correlates with the generalizing algorithm rather than the memorizing solution. This correlation is task-dependent: for natural data, the min-norm interpolator is generally NOT the generalizing algorithm. Grokking on modular arithmetic is a concrete case where the implicit bias aligns with the task's compositional structure.

Necessity of weight decay is an overclaim. Kumar, Bordelon, Gershman, Pehlevan ("Grokking as the Transition from Lazy to Rich Training Dynamics," arXiv 2310.06110, 2023) show grokking in two-layer MLPs without explicit $\ell_2$ weight decay, driven instead by a transition from the lazy (NTK-like) to rich (feature-learning) regime. Other work demonstrates grokking induced by non-$\ell_2$ regularizers (e.g., sparsity penalties, gradient noise) and argues that in some overparameterized settings grokking can appear without any explicit regularization at all, with the implicit bias of SGD providing the compression.

Separately, the $1/\lambda$ scaling is approximate and task-dependent. For some tasks, even strong weight decay does not produce grokking if the generalizing solution requires large weights in specific directions.