RoPE Visualization | Rotary vs Sinusoidal Position Encoding

The idea: put the position into the angle

A transformer has to be told where each token is. RoPE (rotary position embedding) does it by turning the query and the key. It reads them as pairs of numbers, points in a plane, and rotates the pair of a token at position m by the angle m·θ. When a query at m meets a key at n, the angle between them is (n − m)·θ. The dot product only sees that angle, so the score depends on how far apart the tokens are, not on where they are.

One 2-D pair: the query at position 2 is rotated by 2 times 15 degrees, the key at position 5 by 5 times 15 degrees, and the angle between them is 3 times 15 degrees = 45 degrees. On the right, positions 10 and 13 are rotated further, but the angle between them is again 45 degrees. — Rotate q by m·θ and k by n·θ: the angle between them, and so q·k, is (n − m)·θ.

The canvas shows one attention score in a head with only 6 dimensions, so 3 pairs and 3 dials. The query and key carry the same content, the vector (1, 0) in every pair, so everything that changes comes from the positions. (Demo 1: rotate q and k, positions 2 and 5)

Why attention needs positions

Attention compares every query with every key by a dot product and mixes the values by the weights. Nothing in that formula knows the order of the tokens: shuffle the sentence and each token gets the same output, just in another row. "dog bites man" and "man bites dog" would look the same. So the position has to be put into the vectors themselves. See Self-Attention for the attention layer without it.

Sinusoidal encodings: added to the embedding

The original transformer (2017) adds a fixed vector PE(p) to the embedding of the token at position p. Each pair of dimensions holds (sin p·ωᵢ, cos p·ωᵢ) with a different frequency ωᵢ. On the canvas ωᵢ is the same as RoPE's θᵢ and the embedding is x = (1, 0) per pair. The score of two tokens then splits into four parts:

(x + PE(m))·(x + PE(n)) = x·x + x·PE(m) + x·PE(n) + PE(m)·PE(n)

The last part is Σ cos((n − m)·ωᵢ): it only depends on the distance. The middle part does not. It uses m and n themselves, so the same two words at the same distance get a different score in another place:

q at, k at	n − m	RoPE q·k	sinusoidal score
2, 5	3	0.69	5.61
10, 13	3	0.69	5.29
0, 4	4	0.97	4.22
20, 24	4	0.97	5.93

In a real model the query and key are first multiplied by W_Q and W_K, which mixes the four parts further. The network can learn to use the relative part, but nothing forces it to. (Demo 2: same distance, same score)

RoPE: rotate each pair

RoPE leaves the embedding alone. After the query and key are computed it rotates every pair (q₂ᵢ₋₁, q₂ᵢ) of the query at position m by the angle m·θᵢ, with the 2-D rotation matrix:

[q'₁]   [cos mθ  −sin mθ] [q₁]
[q'₂] = [sin mθ   cos mθ] [q₂]

The key at position n is rotated the same way by n·θᵢ. The values are not rotated. Written with complex numbers, a pair is one number q₁ + i·q₂ and the rotation is a multiplication by e^{i·m·θ} (Euler's formula).

Why only the distance survives

A rotation does not change lengths, and the dot product of two vectors is |q|·|k|·cos(angle between them). Rotating q by mθ and k by nθ changes that angle by exactly (n − m)θ. In complex form the score of one pair is

Re[ q·e^{imθ} · conj(k·e^{inθ}) ] = Re[ q·conj(k) · e^{i(m − n)θ} ]

and the positions only appear as m − n. So the query at 2 and the key at 5 score exactly as the query at 10 and the key at 13: cos 180° + cos 45° + cos 11.25° = −1.00 + 0.71 + 0.98 = 0.69. That is also why RoPE works with a KV cache: a cached key was rotated once, for its own position, and never has to change. (Demo 4: far from the start but close together)

Many frequencies, like clock hands

Each pair turns at its own speed, θᵢ = base^(−2i/d). With the real base = 10000 and d = 128 the first pair turns 1 radian (57°) per token and the last one about 0.0001 radian. The canvas uses a short ladder, 60°, 15° and 3.75°, each 4 times slower than the one before.

The fast hand cannot tell distance 6 from 0; the slower hands can.

Like the second hand of a clock, the fast pair repeats: at distance 6 it has gone round once and looks exactly like distance 0. It is good at telling neighbours apart. The slow pairs barely move from one token to the next but keep counting over long distances. Together they give every distance its own pattern. (Demo 3: clock hands, distance 6 fools the fast hand)

Longer contexts: position interpolation

A model trained on 16 positions has only seen distances up to 15, so pair i has only seen angles within ±15·θᵢ: the green arc on each dial. For the fast pairs that arc covers the whole circle, so every angle is familiar. For the slow pair it is only ±56.25°. Put the query at 3 and the key at 30 and the slow pair's angle is 27·3.75° = 101.25°, an angle the model has never seen; real models' attention breaks down there.

The slow pair's dial twice. Left, scale 1: the green trained arc spans plus and minus 56.25 degrees around q, and k at distance 27 sits at 101.25 degrees, outside it. Right, scale one half: distance 27 becomes 13.5 and k sits at 50.6 degrees, inside the arc. — Extrapolation lands outside the trained arc; interpolation squeezes the same distance back inside.

Position interpolation (Chen et al., 2023) multiplies every position by s = L_train / L_new, here 16 / 32 = ½, before rotating. Distance 27 is then rotated like 13.5, and every angle falls inside the trained arcs. The model has to be fine-tuned a little, because neighbours are now only half a position apart, but that is much easier than learning new angles. Later methods such as NTK-aware scaling and YaRN scale the slow pairs more than the fast ones. (Demo 5: distance 27, longer than training; Demo 6: position interpolation)

This page vs a real model

Parameter	This page	Llama 2 7B
head dimension `d`	6 (3 pairs)	128 (64 pairs)
frequencies `θᵢ`	60°, 15°, 3.75°	10000^(−2i/128) rad
trained context	16	4,096
q and k	(1, 0) in every pair	W_Q·x and W_K·x, learned

What the page leaves out

The learned projections W_Q and W_K, the softmax and the values: the page stops at one raw score. Real implementations rotate all heads at once with precomputed cos and sin tables, and some pair dimension i with i + d/2 instead of neighbours, which is the same idea with the dimensions in another order. Learned absolute position embeddings (GPT-2), ALiBi's distance penalty, and the details of NTK-aware scaling and YaRN are left out too.