Drills: Pretraining.
Five questions at a time from the whole volume, until the shape of a pretraining run and what comes out of it are familiar.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
The volume follows one run from its objective to its product: one average loss over packed sequences, curves that show learning and memorising, skills that arrive at different times and look sudden under some metrics, strings stored verbatim when they repeat, facts that depend on particular components, and at the end a base model that continues documents.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- State the pretraining objective and its units.
- Read loss curves and convert losses to bits per byte.
- Separate metric artefacts from real jumps.
- Describe what a base model can and cannot do.
What to carry forward
Pretraining minimises one average over every token (plate VIII.1), packed into full sequences (plate VIII.2). Its progress is read from two curves that part when data repeats (plate VIII.3), in units that only compare fairly across tokenisers as bits per byte (plate VIII.4). Inside the average, skills arrive at different times (plate VIII.5), and some of their apparent suddenness belongs to the metric (plate VIII.6).
What the run leaves in the weights is both general and particular: strings repeated often enough are memorised (plate VIII.7), facts depend on identifiable components without living at a single address (plate VIII.8), and the whole is a base model that answers only when the prompt makes an answer the likely continuation (plate VIII.9). Vol. IX turns to the data that decides all of this.
Terms
- Bits per byte
- Code length per byte of text; tokeniser-free.
- Epoch
- One pass through the training data.
- Canary
- A planted random string.
- Base model
- A pretrained model that continues documents.