What Prediction Teaches.
Nobody tells a language model what a plural is or what carbon combines with. It learns whatever lowers the loss — cheapest lessons first.
A language model is never taught grammar, vocabulary or facts. It is only asked, over and over, to put probability on the next token. But whatever helps with that question gets learned along the way. Agreement between a subject and its verb is worth a lot of loss and is visible in almost every sentence, so it is learned early. Which words go together in a subject — carbon dioxide, the speed of light — is learned next. Facts that appear rarely, such as which city is the capital of which country, help with only a few predictions in the whole corpus and need a much larger model and much more data.
One way to find out what a model has learned is to probe it: give it a prompt, compare the probability it assigns to a right continuation and a wrong one, and see which it prefers. This plate probes the course model micro — 1.9 million parameters, trained only on Curionode’s course prose — in three kinds of question.
Below: twelve probes in three groups — grammar, words that belong together, and facts — plus one of your own. Each bar is how many times more likely the model finds the right continuation than the wrong one, as a power of ten; green bars are right, red are wrong.
- Probe a model by comparing the probability of two continuations.
- Explain why prediction teaches grammar before facts.
- Relate what a model knows to what is common in its training data.
- Say which conclusions about a small model carry over to large ones.
Cheap lessons first
The model gets 4 of the four grammar probes and 4 of the four word-association probes right, often by a wide margin: after The cells it prefers are to is by a factor of 10 to the 1.6, and after carbon it prefers dioxide to apple by 10 to the 5.7. It gets 0 of the four fact probes right; on the capital of France its preference is −0.42 — essentially a coin toss between two rare names. Agreement and collocations recur in nearly every paragraph of its corpus, so learning them lowers the loss a great deal. A fact such as a capital city is useful in a handful of sentences out of millions, and a model this small, trained on this little text, has not found it worth storing.
Large models trained on trillions of tokens of varied text do learn huge numbers of facts this way, and many more abstract skills, because every one of them lowers the loss somewhere. The order in which they appear during training — and whether some appear suddenly — is the subject of plates VIII.5 and VIII.6. Two cautions carry over from this tiny example. A probe only tests the two continuations it compares; and a model that prefers the right answer is not thereby reasoning about it — it may simply have seen the phrase. Vol. VIII asks where such knowledge lives in the weights, and Vol. XXVIII how to read it out.
Terms
- Probe
- A test that compares the probabilities a model gives to chosen continuations to find out what it has learned.
- Agreement
- The grammatical matching of words, such as a plural subject with a plural verb.
- Collocation
- Words that habitually occur together, like carbon dioxide.
- Emergent skill
- A capability that appears only past some scale of model, data or training; plate VIII.6 measures it.