Vol. VI · Positions and Long Context Plate 3 of 10RoPE, rotary position embedding, relative position, rotation, frequenciesA

Rotary Position Embedding.

Turn each query and key by an angle set by its position. When two of them meet in a dot product, only the difference of their angles is left.

Rotary position embedding, RoPE (Su and colleagues, 2021), adds nothing to the token vectors. Instead, inside attention, it splits each query and key into pairs of coordinates and rotates each pair by an angle m θ_i, where m is the token’s position and θ_i a frequency that differs from pair to pair. A rotation preserves length, so the content of the vector is untouched; only its direction in each plane encodes position.

The point is what happens in the dot product. Rotating the query by mθ and the key by nθ and then taking their dot product is the same as rotating by (m − n)θ alone: the absolute positions cancel and only their difference remains. Attention scores therefore depend on token content and relative distance, which is what language mostly cares about. Llama, Qwen, Mistral and most open models since 2023 use RoPE.

Below: left, one query and one key in a single 2-D plane, rotated by their positions; move both positions together and the angle between them, and so their dot product, does not change. Right, the attention score of a fixed query–key pair, summed over 64 frequency pairs, as their distance grows.

What you will learn

Describe RoPE as position-dependent rotations of queries and keys.
Show that the dot product depends only on relative position.
Explain the role of different frequencies.
See the long-range decay of RoPE scores.

Workshop

Left: a query (red) at position m and a key (blue) at position n in one plane, after rotation by mθ and nθ (θ = 0.3). Right: the score Σᵢ qᵢ · R((m − n)θᵢ) kᵢ of one random query–key pair over 64 frequency pairs, against the distance m − n.

Positions that cancel

With the query at position 7 and the key at position 3, their dot product after rotation is −0.4542. Shift both by 57 — positions 64 and 60 — and it is still −0.4542, because each rotation adds 57θ to both angles and the difference is unchanged. In a real model each head has 32 to 64 such planes, rotating at frequencies from about one radian per position down to one ten-thousandth, so fast planes resolve neighbouring tokens and slow planes distinguish positions thousands apart.

Summed over 64 planes, the score of a fixed query–key pair is 112.41 at distance 0 and drifts towards smaller values with distance (44.96 at 300) as the fast planes stop lining up — a built-in tendency to favour nearby tokens that the model can override with content. RoPE needs no extra parameters and works with the KV cache, because each key is rotated once by its own position. What it does not do by itself is generalise to positions far beyond training: the slow planes reach angles they never saw. Plates VI.5 and VI.6 measure that and the fixes.

The frequency base 10 000 was inherited from the 2017 sinusoids; models designed for long contexts raise it — Llama 3 uses a base of 500 000 — so that the slowest planes turn less over a long text.

Terms

RoPE: Rotary position embedding: rotate query and key pairs by angles proportional to position.
Relative position: The distance between two tokens, m − n, rather than their absolute indices.
Frequency: The angle per position θᵢ of one coordinate pair; RoPE uses many, geometrically spaced.
Base: The constant (often 10 000) that sets RoPE’s range of frequencies.

RoPE rotates queries and keys by their positions, so their dot product depends only on the distance between them — relative position, built into attention.

Sources

Su, Lu, Pan, Murtadha, Wen and Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding”, arXiv:2104.09864, 2021.