Vol. V · The Transformer Plate 9 of 10training a GPT, live training, loss curve, sampling, course model nano, computeL

Training It.

Press the button and a GPT learns, in your browser, from nothing. Then see what eight thousand GPU steps of the same recipe produce.

Training is the loop of Vol. III applied to the model of this volume: take a batch of text, run the forward pass, compute the cross-entropy at every position, backpropagate, take an AdamW step, repeat. The workshop runs that loop live, in a background thread of your browser, on a character-level GPT of a few tens of thousands of parameters, for a few hundred steps on a slice of the course corpus. It is slow — a laptop processor manages around a billion floating-point operations a second at this — but every number is computed in front of you.

Beside it is the course model nano: the same kind of character-level GPT, two layers of width 64, 113 thousand parameters, trained for 8 000 steps on 29 million characters on a GPU. Its loss curve was recorded, and it runs in the page, so its samples are live. The gap between the two is not a different algorithm; it is several thousand times more compute.

Below: left, the live run — training loss per step and held-out loss at intervals — and a sample from the model it produced. Right, nano’s recorded curve and a sample from nano. Change the live settings to train again; it takes a few seconds.

What you will learn

Run a training loop and read its loss curve.
Sample from a model you have just trained.
Compare a short run with a long one of the same design.
Relate the quality of samples to loss and compute.

Workshop

Left: the live run in your browser (training loss blue, held-out red) and a 60-character sample from it after “The ”. Right: the recorded run of nano (8 000 GPU steps) and a sample from nano, running in the page.

Thousands of times the compute

The live model starts at 4.838 nats per character — a uniform guess over its alphabet — and after 100 steps of 8 sequences reaches 3.447 on held-out text. Its samples are letters in roughly the right proportions with spaces between them — no words yet. nano, with 113,792 parameters and 8,000 steps on far more data, reaches 1.492, and its samples are recognisably English, with real words, punctuation and a grasp of the course’s subjects, though not of sense. nano used about 5,380 times as many floating-point operations as the live run.

Nothing else separates the two: same architecture family, same objective, same optimiser. That is the lesson Part Two develops — loss falls smoothly and predictably with compute (Vol. X), and so does the quality of what a model can do. It is also a lesson about where the work of a real training run goes: not into a cleverer loop than this one, but into the data that feeds it (Vol. IX), the stability that keeps it going (Vol. XI) and the machinery that makes it possible to spend a hundred million times more compute than nano did (Part Three).

The live run uses js/tensor.js, whose losses and gradients are checked against PyTorch on the same weights; on a laptop processor in a browser it reaches roughly a billion floating-point operations a second, about a ten-thousandth of what the GPU that trained nano sustains at this size.

Terms

Training loop: Batch, forward, loss, backward, optimiser step, repeated.
Live training: Training run in the page as you watch, here in a background worker thread.
Sample: Text generated from a model by sampling its next-token distribution.
Compute: Floating-point operations spent; here about 6 × parameters × tokens.

Training a GPT is the same loop at every scale; the live model and nano differ by thousands of times the compute, and it shows in both their losses and their samples.