Drills: Language Models.
Five questions at a time from the whole volume, until predicting, scoring and sampling feel routine.
This plate has no new material. It draws rounds of five from the questions and predictions of the nine plates before it, in a different order each time, with the reasoning shown after each answer and the plate it came from.
Work through rounds until you can answer without hesitating. The volume’s ideas — a distribution over the next token, the chain rule, surprisal and bits per byte, greedy and sampled generation, sparse counts and generalising networks — are used on almost every later plate.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five from the pool.
- Recall the definitions of the volume without looking them up.
- Convert between bits, nats, perplexity and bits per byte quickly.
- Predict the effect of n, smoothing and temperature.
- Find weak spots to revisit.
What to carry forward
Four ideas from this volume are used constantly from here on. A language model is a function from a text to a probability distribution over the next token (plate I.1). The probability of a text is a product of those distributions, a sum of surprisals in logarithms (plate I.3). A model is graded by its average surprisal on held-out text, in bits per byte when tokenisers differ (plate I.4). And how the distribution is turned into text is a separate choice from the model itself (plate I.5).
Two more explain why the field moved from counting to networks: long contexts are sparse, so tables cannot generalise (plate I.6), while models that represent tokens as vectors can (plate I.7). The next volume looks at the first thing a model does with a text — cutting it into tokens.
Terms
- Distribution
- Probabilities over every possible next token, summing to one.
- Cross-entropy
- Average surprisal per token on held-out text.
- Autoregressive
- Generating by feeding each output back as input.
- Generalisation
- Predicting well on contexts never seen in training.