Position
Lesson 5 of 8
5Position
The big pictureAttention does not see order. "dog bites man" and "man bites dog" are to it exactly the same set of words. So we inject position into the vectors somehow, and today's most common way is called RoPE: rotating the query and key by an angle that depends on each one's position.
What the picture shows: These are two real columns of the sinusoidal table from the lesson: the first swings fast like a second hand, the second moves slowly like an hour hand. Together they give each position a fingerprint, and nearby positions get similar fingerprints.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Attention alone does not know order: "the boy ate the apple" and "the apple ate the boy" look like the same bag of words to it. So we give each word its seat number, like a theatre ticket, and the model knows who came before whom.
Positional encoding is either absolute (sinusoidal or learned, added to the embedding) or relative. The most common today is RoPE, rotating Q and K by an angle that follows position, so their product depends on the distance between tokens and context can be stretched further.
Before You Start
👀Attention
You know attention compares every query with every key, and nothing in the equation mentions word order.
🧱The transformer block
You know where the query and key enter the block, because RoPE works on them directly.
The Core Idea
In this lesson you first prove that attention cannot see order, then see three ways to add position: an added vector, the sinusoidal encoding, then RoPE.
1Attention is blind to order
Attention treats a sentence as a set of words, not a sequence. The two sentences in the picture have the same words in reverse order.
import numpy as np
rng = np.random.default_rng(0)
d = 4
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))
def attend(x):
s = (x @ Wq) @ (x @ Wk).T / np.sqrt(d)
w = np.exp(s - s.max(1, keepdims=True))
w /= w.sum(1, keepdims=True)
return w @ (x @ Wv)
emb = {w: rng.normal(size=d) for w in ["dog", "bites", "man"]}
a = np.stack([emb[w] for w in ["dog", "bites", "man"]])
b = np.stack([emb[w] for w in ["man", "bites", "dog"]])
out_a, out_b = attend(a), attend(b)
print("output for 'dog' in sentence A equals 'dog' in sentence B:",
bool(np.allclose(out_a[0], out_b[2])))
print("same set of outputs, only reordered:",
bool(np.allclose(np.sort(out_a, 0), np.sort(out_b, 0))))
The output for dog is the same whether it is the subject at the start or the object at the end. Attention gives the same outputs, only reordered. So the model cannot tell who bit whom. This is called permutation equivariance.
2The first fix: add the position
What we need is for the same word to look different at two positions. Look at the picture: "dog" at position 0 enters the first block as a different vector from "dog" at position 2.
The simplest idea: a vector for each position (0, 1, 2, ...) added to the word embedding before the first block. Either learned as a second embedding table, as in GPT-2 and in the transformer we will build, or computed with sine functions as in the original transformer.
3The sinusoidal encoding
Instead of a table we learn, each position's vector can be computed with sines and cosines at different frequencies. It has no parameters, and the original transformer used it.
import numpy as np
def sinusoidal(T, d):
pos = np.arange(T)[:, None]
i = np.arange(0, d, 2)[None, :]
angle = pos / 10000 ** (i / d)
pe = np.zeros((T, d))
pe[:, 0::2] = np.sin(angle)
pe[:, 1::2] = np.cos(angle)
return pe
np.set_printoptions(precision=2, suppress=True)
pe = sinusoidal(T=6, d=8)
print(pe + 0.0)
near = float(pe[2] @ pe[3])
far = float(pe[2] @ pe[5])
print(f"position 2 vs 3: {near:.2f} position 2 vs 5: {far:.2f}")
Each row is a position; the first columns change fast and the last ones very slowly, like clock hands: seconds, minutes, hours. And positions 2 and 3 are more similar (3.54) than 2 and 5 (1.96), so nearby positions have similar vectors and the model can learn "closeness".
4RoPE: the score depends on distance
Relative position is more useful than absolute: "three words before me" matters more than "position 7". RoPE rotates the query by its position's angle and the key by its own.
The same query and the same key at different positions. We look at the score when the gap between them is fixed.
import numpy as np
def rope(v, pos, base=10000):
d = len(v)
theta = pos / base ** (np.arange(0, d, 2) / d)
cos, sin = np.cos(theta), np.sin(theta)
x, y = v[0::2], v[1::2]
out = np.empty(d)
out[0::2] = x * cos - y * sin
out[1::2] = x * sin + y * cos
return out
q = np.array([1.0, 0.5, -0.3, 0.8])
k = np.array([0.2, -1.0, 0.7, 0.4])
for m, n in [(3, 1), (10, 8), (105, 103), (5, 1)]:
score = rope(q, m) @ rope(k, n)
print(f"query at {m:>3}, key at {n:>3}, gap {m - n}: score {score:+.4f}")
print("length unchanged by rotation:", bool(np.isclose(np.linalg.norm(rope(q, 50)), np.linalg.norm(q))))
The pairs (3, 1), (10, 8) and (105, 103) all have a gap of 2 and gave exactly the same score, even though their absolute positions are completely different. A gap of 4 gave a different score. This property is why RoPE spread: the model learns relative relations that hold anywhere in the text.
Look at the picture: rotation does not change a vector's length, so it does not spoil the information it carries. The score between them depends only on the angle between them.
How It Works, Step by Step: how RoPE puts position into attention
Split the query and key dimensions into pairs, each pair a point in a plane.
Rotate each pair by an angle equal to the position times that pair's own frequency, fast for the first pairs and slow for the last.
Do this to the query at its position m and to the key at its position n, without changing either one's length.
In the dot product only the angle m − n remains, so the score depends on the distance between the two words alone.
Going deeper
That is why most modern open models (Llama, Qwen, Mistral and others) adopted RoPE. Still, when you see a model promising a million-token context, remember that "accepts" does not mean "uses well". Research found models recall the start and end of a context better than the middle, which you will measure yourself in the context engineering lesson.
Worked example
First let us prove the problem is real. Two sentences with the same words in reversed order, and attention with no position information at all.
max_position_embeddings in the config file teaches the model no new positions. Real extension methods (such as RoPE scaling) usually need extra training on long texts, and are measured afterwards on a retrieval task from the middle of the context.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| Attention knows the order because words go in one after another. | All words are computed together as matrix products. In the example "dog bites man" and "man bites dog" gave the same outputs, only reordered. |
Raising max_position_embeddings extends a model's context. | Changing the config teaches the model no new positions. Real extension usually needs extra training on long texts, then measuring. |
| A model that accepts a million tokens uses all of them well. | "Accepts" does not mean "uses well". Models recall the start and end of a context better than the middle. |
Try it yourself
What happens to attention outputs (with no position encoding) if you reverse the word order?
Every word sees the same set of words whatever their order, so its output does not change. That is why we need position information.
In RoPE, what does the attention score between a query at position m and a key at position n depend on?
Rotating by angle m and multiplying with a key rotated by n equals rotating by their difference. That is what you saw in the example: the same score for every pair with a gap of 2.
Quick review cards
Check Your Understanding
Why does a learned position table not work for a model trained on 2,048 tokens and then given 4,000?
Because the table has only 2,048 rows, so there is no vector for position 2,049 at all. Even if we extend it, the new rows are untrained. RoPE computes the angle for any position, so it technically works past the training length, though quality drops without scaling.
In RoPE, why does the rotation not spoil the information the vector carries?
Because rotation changes direction and not length, as the example confirmed with True. And the angle itself is known from the position, so nothing is lost.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Done with this lesson?
Sign in to save your progress.