Sparse Autoencoders for Interpretability

A neural network with activation dimension $d$ can represent $F \gg d$ features by encoding each feature as a direction in $\mathbb{R}^d$. Elhage et al. 2022's toy-model analysis shows that in certain regimes the number of representable features scales roughly as $F \sim d / p$, where $p$ is the per-feature activation probability. In the very-sparse limit ($p \to 0$) $F$ can scale even faster, approaching the Kabatyanskii-Levenshtein bound of $\exp(\Omega(d \varepsilon^2))$ nearly-orthogonal directions (pairwise inner product $\leq \varepsilon$). The $d/p$ approximation should be read as an informal scaling observation from a toy model, not a tight theoretical bound. See Scherlis, Sachan, Jermyn, Benton, Shlegeris 2022 ("Polysemanticity and Capacity in Neural Networks", arXiv:2210.01892) for a more rigorous capacity analysis.

The features are packed as nearly-orthogonal directions. The interference between any two features $i, j$ is proportional to $|\langle f_i, f_j \rangle|$, which is small but nonzero. The network tolerates this interference because, by sparsity, it is rare for two interfering features to be active simultaneously.

In $\mathbb{R}^d$, you can fit exactly $d$ orthogonal directions. But if you allow small angles between directions (small interference), you can fit exponentially more nearly-orthogonal directions. This is the Johnson-Lindenstrauss phenomenon applied to feature representation. The network exploits sparsity: if two features that interfere are rarely active at the same time, the interference rarely causes errors.

Superposition explains why looking at individual neurons is misleading. The meaningful unit is not a neuron axis but a feature direction. This is why you need sparse autoencoders: they recover the feature directions from the superposed representation, producing a dictionary of monosemantic features from a polysemantic activation space.

The hypothesis is well-supported empirically in toy models and small transformers. It is less clear how precisely it applies to frontier models with billions of parameters, where the relationship between features, neurons, and directions may be more complex. The toy model results (Elhage et al., 2022) demonstrate superposition convincingly in controlled settings, but scaling to GPT-4-class models is ongoing work.

A sparse autoencoder learns an encoder $f: \mathbb{R}^d \to \mathbb{R}^M$ and decoder $g: \mathbb{R}^M \to \mathbb{R}^d$ that minimize:

$\mathcal{L} = \|x - g(f(x))\|_2^2 + \lambda \|f(x)\|_1$

where:

• $f(x) = \text{ReLU}(W_{\text{enc}}(x - b_{\text{dec}}) + b_{\text{enc}})$, the encoder with pre-centering
• $g(z) = W_{\text{dec}} z + b_{\text{dec}}$, the linear decoder
• $W_{\text{enc}} \in \mathbb{R}^{M \times d}$, $W_{\text{dec}} \in \mathbb{R}^{d \times M}$
• $\lambda > 0$ controls the sparsity-reconstruction tradeoff

The decoder columns $W_{\text{dec}}[:, j]$ are the learned feature directions. The encoder activations $f(x)_j$ indicate how much feature $j$ is present in input $x$.

The SAE is solving dictionary learning: find a set of $M$ basis vectors (the decoder columns) such that each activation $x$ can be well-approximated as a sparse linear combination of these basis vectors. The L1 penalty ensures that only a few features are active for any given input, which is what makes the features interpretable (each feature corresponds to a specific, identifiable concept).

The overcomplete factor $M/d$ is typically 4x to 64x. A 64x SAE on a 4096-dimensional activation space learns $\sim$262,000 feature directions.

The trained decoder columns become the interpretable feature dictionary. Each column represents a direction in activation space that corresponds to a specific concept, behavior, or pattern. Researchers can then study which features activate on which inputs, how features compose, and how the network uses features to produce outputs. This is the primary tool for understanding what transformer layers compute.

The sparsity-reconstruction tradeoff is fundamental. Higher $\lambda$ produces sparser (more interpretable) features but worse reconstruction. Lower $\lambda$ produces better reconstruction but features that are less cleanly monosemantic. There is no universal "correct" $\lambda$. Additionally, SAEs can learn dead features (dictionary elements that never activate) and feature splitting (a single concept split across multiple dictionary elements). Both require careful training techniques to mitigate.