Vol. V · The Transformer Plate 8 of 10GPT, forward pass, step by step, embeddings, blocks, logits, course modelA

A GPT in the Page.

A real language model, 1.9 million parameters, running in your browser. Stop it at any stage and look inside.

Everything in Vols. I–V comes together in micro, the course model this page runs. Its forward pass is: split the prompt into tokens with the course tokeniser; look each id up in the 8 192 × 128 embedding table and add the learned vector for its position; pass the sequence through four blocks of norm–attention–add and norm–MLP–add; apply the final norm; multiply by the transposed embedding table to get 8 192 logits for the last position; and take a softmax. About 1.9 million multiply–adds per token.

The code that does this in your browser (js/lm.js) is held to PyTorch on the same weights: the logits agree to better than a thousandth. The workshop lets you stop the forward pass at any stage and see the 128 numbers that represent the last token there, together with what the model would predict if it stopped at that point.

Below: the prompt as tokens; the 128-number vector for the last token at the chosen stage, drawn as an 8 × 16 grid (blue positive, red negative); and the top predictions read from that stage. At the last stage, these are micro’s actual predictions.

What you will learn

Trace a prompt through every stage of a GPT.
Read the vector that represents a token at each stage.
See the prediction form across stages.
Count the work of one forward pass.

Workshop

Top: the prompt in tokens. Middle: the last token’s 128-number vector at the chosen stage as an 8 × 16 grid. Bottom: the top predictions at that stage (the logit lens before the end; the real output at the final stage).

Every number, visible

micro has 1,853,696 parameters. A forward token costs it about 3.67 million floating-point operations — twice the non-embedding parameters plus the logits — so reading the 10-token prompt takes about 36.7 million, a few milliseconds of a laptop processor. At the final stage the top prediction has probability 9.5%. Step back through the stages and the vector for the last token changes from block to block as attention brings in the rest of the prompt and the MLPs transform it; the grid shows a few dimensions growing much larger than the rest, a common feature of trained Transformers sometimes called outlier features (plate XXIV.1 explains why they make quantisation hard).

Nothing here is a simulation of a language model; it is one, small and weak but complete. Production models differ in scale and in details — rotary positions, RMSNorm, gated MLPs, grouped-query attention, a larger tokeniser — that Vols. VI and VII introduce, but a reader who can follow this forward pass can follow theirs. The next plate trains such a model.

micro’s weights ship as 1.9 megabytes of 8-bit integers with a scale per row, dequantised when the page loads; the oracle compares the page’s logits with PyTorch’s on the same dequantised weights, so quantisation does not hide a disagreement.

Terms

Forward pass: Running a model from input tokens to output probabilities.
Course model: One of the small models trained for this course; micro has 1.9 million parameters.
Dequantise: Turn stored low-precision integers back into floating-point weights.
Outlier features: A few dimensions of the residual stream with much larger values than the rest.

A GPT is tokens → embedding + position → L blocks → norm → logits → softmax, and this one runs, inspectably, in the page.