Predicting the Next Token.
Give a model the start of a sentence and it does not answer with a word: it answers with a probability for every word it knows.
Type the start of a sentence into the workshop below. The course model reads it — first chopped into tokens, pieces of words that Vol. II explains — and returns a list as long as its vocabulary: for each of its 8 192 tokens, the probability that it comes next. That list is a probability distribution; its numbers are positive and add up to one. A language model is nothing more than a function from a text to such a distribution.
It sounds too modest to matter. But to put a high probability on the right continuation of “The heart pumps blood through the”, a model has to have absorbed something about hearts; to continue “1, 2, 3, 4,” it has to have noticed counting. Every capability of a large model is reached through this one question, asked billions of times during training: what comes next?
Below: the prompt as the model reads it, one box per token, and the ten tokens it thinks most likely to come next with their probabilities. Edit the prompt; switch to the character-level model and the “tokens” become single letters.
- Describe a language model as a function from text to a next-token distribution.
- Read a probability distribution: its largest entries, its total and its spread.
- Measure the uncertainty of a prediction as entropy, in bits.
- Explain why next-token prediction forces a model to learn about the world.
A distribution, not a word
For the prompt above, the course model micro gives its single most likely next token a probability of 7.0%, and its ten most likely together 29.7%; the rest — more than eight thousand tokens — share what is left. The entropy of the whole distribution is 8.07 bits, which is as uncertain as a fair choice among about 269 equally likely tokens. Replace the prompt with 1, 2, 3, 4, and the distribution sharpens: the entropy falls to 3.94 bits, because counting is a pattern the model has seen many times.
Switch to nano, a model that reads one character at a time, and the same prompt gives an entropy of 0.42 bits over a much smaller alphabet: predicting the next letter is easier than predicting the next word-piece, but each prediction says less. Both are language models in exactly the same sense. Generating text (plate I.5), scoring it (plate I.3) and answering questions (Part Four) are all ways of using this one distribution; the rest of the course is about how such a function is built, trained, made useful and served.
Terms
- Language model
- A function that assigns a probability to every possible next token given the text so far.
- Token
- The unit a model reads and predicts: a character, a byte, a word or a piece of a word.
- Vocabulary
- The fixed set of tokens a model can read and predict; its size is the length of each distribution.
- Entropy
- The average surprise of a distribution, in bits: zero when one token is certain, larger when the probability is spread out.