Positional Encoding

For any fixed offset $k$, the positional encoding at position $\text{pos} + k$ is a linear function of the encoding at position $\text{pos}$. Specifically, for each frequency band $i$:

$\begin{pmatrix} \sin(\omega_i(\text{pos}+k)) \\ \cos(\omega_i(\text{pos}+k)) \end{pmatrix} = \begin{pmatrix} \cos(\omega_i k) & \sin(\omega_i k) \\ -\sin(\omega_i k) & \cos(\omega_i k) \end{pmatrix} \begin{pmatrix} \sin(\omega_i \cdot \text{pos}) \\ \cos(\omega_i \cdot \text{pos}) \end{pmatrix}$

where $\omega_i = 1/10000^{2i/d}$.

The matrix is a rotation matrix $R_{\omega_i k}$. So shifting position by $k$ is equivalent to rotating each 2D frequency component by $\omega_i k$.

The sinusoidal encoding represents position as a point on a collection of circles (one per frequency band). Moving forward by $k$ positions rotates the point on each circle by a frequency-dependent angle. Since rotation is a linear operation, the model can learn to detect relative position offsets using linear projections. no nonlinear computation needed.

This linear relationship means that a linear attention head can, in principle, learn to attend based on relative position. The rotation structure of the sinusoidal encoding is the direct precursor to RoPE, which takes this idea further by applying rotations to queries and keys rather than adding vectors to embeddings.

In practice, the additive sinusoidal encoding works poorly for long sequences. Because position information is mixed into the token embedding by addition, the model must disentangle content and position information. For long sequences, the position signal becomes a small perturbation on the content signal, making it hard for the model to use. Modern LLMs have abandoned sinusoidal encoding in favor of RoPE.

The attention logit between position $m$ (query) and position $n$ (key) under RoPE depends only on the relative distance $m - n$:

$\tilde{q}_m^\top \tilde{k}_n = q^\top R_m^\top R_n k = q^\top R_{n-m} k$

This follows from the orthogonality of rotation matrices: $R_m^\top R_n = R_{n-m}$.

Explicitly, the attention logit decomposes into $d/2$ terms, one per frequency band:

$\tilde{q}_m^\top \tilde{k}_n = \sum_{i=0}^{d/2-1} \left[(q_{2i} k_{2i} + q_{2i+1} k_{2i+1})\cos((m-n)\theta_i) + (q_{2i+1} k_{2i} - q_{2i} k_{2i+1})\sin((m-n)\theta_i)\right]$

Each term depends on the relative position $m - n$ and the frequency $\theta_i$.

Rotating the query by angle $m\theta$ and the key by angle $n\theta$, then taking their dot product, gives a result that depends on the angle difference $(m-n)\theta$. This is exactly how relative position should work: the attention between tokens 5 and 3 should be the same as between tokens 105 and 103, because the relative offset is the same.

Each frequency band captures a different scale of relative position: high-frequency bands distinguish nearby positions, low-frequency bands distinguish distant positions. Together, the $d/2$ frequency bands provide a rich encoding of relative distance.

RoPE is used in Llama, Mistral, Qwen, and virtually all modern open-source LLMs. GPT-NeoX-20B (Black et al. 2022, arXiv 2204.06745) was the first large-scale decoder-only LM to use RoPE, predating Llama. Its success comes from three properties: (1) it naturally encodes relative position without any additive modification to the embeddings, (2) base RoPE adds zero trainable parameters (variants like YaRN and learned RoPE do add small numbers of trainable parameters, so this property is not universal across the RoPE family), and (3) it composes cleanly with the attention mechanism because the position information lives in the phase of the query-key dot product rather than in the amplitude of the embeddings.

Production models often increase the RoPE base to improve long-context behavior: Llama 3 (Dubey et al. 2024) uses base $b = 500{,}000$, and Mistral-7B uses $b = 1{,}000{,}000$. A larger base stretches the low-frequency wavelengths so that longer absolute positions still correspond to angles seen in training.

RoPE's context length extrapolation is limited. The base frequencies $\theta_i = 10000^{-2i/d}$ span a wide range: $\theta_0 = 1$ is the highest frequency (shortest period $2\pi$), and $\theta_{d/2-1} = 10000^{-(d-2)/d}$ is the lowest frequency (longest period $2\pi / \theta_{d/2-1}$, which for $d=128$ is on the order of $6 \times 10^4$ positions). The real failure at test-time lengths beyond training is not literal wrap-around of angles past $2\pi$, but distribution shift: the model encounters angle combinations $(m\theta_0, m\theta_1, \ldots)$ that never appeared during training, and attention heads trained on one angular regime behave unpredictably on another. Techniques like NTK-aware scaling, YaRN (Peng 2023, arXiv 2309.00071), and dynamic NTK interpolation modify the frequency base to keep test-time angles inside the training distribution without full retraining.

ALiBi (Press et al., 2022) does not modify the input embeddings or the Q/K projections at all. Instead, it adds a position-dependent bias directly to the attention logits:

$\text{Attention}_{\text{ALiBi}} = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} - \lambda \cdot |m - n|\right)V$

where $\lambda > 0$ is a head-specific slope and $|m - n|$ is the absolute distance between positions $m$ (query) and $n$ (key). Press et al. 2022 set the slopes as a geometric sequence starting at $2^{-8/H}$ with common ratio $2^{-8/H}$: for head $h \in \{1, \ldots, H\}$, the slope is $m_h = 2^{-8h/H}$.

The linear bias penalizes distant tokens, making attention prefer nearby positions. Different heads use different slopes, so some heads attend locally and others more globally.

ALiBi says: "All else being equal, prefer to attend to nearby tokens." The linear penalty $\lambda|m-n|$ acts as a soft window. Tokens that are far away must have very high content similarity (large $q^\top k$) to overcome the distance penalty. This is a reasonable inductive bias: in natural language, nearby words are usually more relevant than distant ones.

ALiBi was designed specifically for context length extrapolation. Because the position information is a simple additive bias rather than a modification to the embeddings, the scheme generalizes smoothly to positions never seen during training: position 20000 just gets a proportionally larger penalty. In the original perplexity experiments of Press et al. 2022, ALiBi extrapolated better than sinusoidal and learned positional encodings. Kazemnejad et al. 2023 (arXiv 2305.19466) later showed that the picture is task-dependent: on some length-generalization benchmarks (algorithmic tasks, certain reasoning tasks), ALiBi does not generalize better than other schemes, and in some cases no positional encoding at all outperforms ALiBi. Treat the "extrapolates" claim as "extrapolates on perplexity for natural language, sometimes fails elsewhere."

ALiBi assumes that attention should decay with distance, which is not always true. Tasks requiring long-range dependencies (e.g., matching an opening bracket with a closing bracket 10000 tokens later) are penalized by the linear bias. RoPE does not have this bias: tokens at any distance can attend to each other based purely on content similarity. This is one reason RoPE has become more popular than ALiBi for general-purpose LLMs.