Vol. VI · Positions and Long Context Plate 2 of 10absolute position embedding, sinusoidal encoding, learned positions, position vectorsA

Absolute Positions.

Give position 37 its own vector and add it to whatever token sits there. Simple, effective, and stuck at the length it was trained for.

The simplest way to tell a model where a token is: keep a vector for every position, 0 to T − 1, and add it to the token’s embedding before the first block. The 2017 Transformer used fixed vectors made of sines and cosines at geometrically spaced frequencies, so that each position has a unique pattern and nearby positions have similar ones. GPT-2 and BERT simply learned the table, one row per position, by gradient descent like everything else; micro does the same, for its 128 positions.

Both work, and both have the same limitation: a learned table has no row for position T, and even sinusoids, which could be computed for any position, generalise poorly beyond the lengths seen in training, because the network has only learned to read the patterns it saw. The workshop compares the sinusoidal table with micro’s learned one by asking how similar each position’s vector is to every other’s.

Below: similarity between the position vectors of positions 0 to 63, for the sinusoidal scheme (left) and micro’s learned positions (right). Bright near the diagonal means that nearby positions have similar vectors.

What you will learn

Describe absolute position embeddings, fixed and learned.
Build the sinusoidal table.
Compare the structure of learned and fixed position vectors.
Explain why absolute positions do not extend past the training length.

Workshop

Cosine similarity between every pair of positions 0–63: sinusoidal position vectors of width 128 (left) and micro’s learned position vectors (right).

A vector per position

The sinusoidal table is built, not learned, and its similarity pattern is a smooth band: neighbouring positions are nearly identical and similarity falls off and oscillates with distance. micro’s learned table came out of training with a related structure — neighbouring positions have an average similarity of 0.986, positions 30 apart only 0.402 — because nearby positions play nearly the same role in language, so gradient descent pushed their vectors together. Nobody told it that position 12 is next to position 13.

The learned table works well inside its range, which is why GPT-2, BERT and GPT-3 used one. Its limit is absolute: micro has 128 rows, so it simply cannot represent a 129th position, and a model whose training sequences were all 64 tokens long has never trained rows beyond 64 even if they exist. Plate VI.5 measures what happens past the training length; the schemes of the next two plates — rotations and linear biases — were designed to make relative distance, not absolute position, what the model sees.

The sinusoidal frequencies run from one cycle per 2π positions to one per 10 000 · 2π; Vaswani and colleagues chose them so that the encoding of position t + k is a linear function of the encoding of t, hoping the model would learn to attend by relative offset.

Terms

Absolute position embedding: A vector for each position index, added to the token embedding.
Sinusoidal encoding: Fixed position vectors of sines and cosines at geometrically spaced frequencies.
Learned positions: A trained table with one row per position.
Training length: The longest sequence the model saw during training.

Absolute schemes add a vector per position to each token; neighbouring positions end up alike, and nothing exists — or is trained — past the training length.