Vol. VIII · The Pretraining Run Plate 9 of 10base model, continuation, prompting, framing, document completion, instruction following, few-shotDE

What a Base Model Is.

Pretraining produces a document continuer — knowledgeable, but answering only when the prompt makes an answer the likely next text.

The model at the end of pretraining is called a base model. It has one skill: given text, predict what comes next, as the training documents would have it. It does not know it is being asked anything. Type a question and it may answer, or continue with another question, or start a new document, depending on what followed such questions in its training data.

The knowledge is nevertheless inside, and the prompt decides whether it comes out. The workshop asks micro four things it knows, each framed three ways: as the start of a statement (“The heart pumps”), as a bare question (“Question: What does the heart pump? Answer:”), and as a question followed by the start of the answer sentence. For each framing it reports the probability and rank of the right word, and samples a continuation. Turning a base model into an assistant that answers questions is the work of Part Four.

Below: left, the probability of the right word under each framing, for the selected fact. Middle, micro’s ten most likely next tokens under the selected framing. Right, a sampled continuation.

What you will learn

Define a base model.
Show that framing changes what a base model produces.
Measure knowledge as the probability of the right continuation.
Explain why instruction tuning is needed.

Workshop

micro, four facts it knows, each in three framings. Left: probability of the right next word (log scale), with its rank among 8 192 tokens in the label. Middle: top ten next tokens for the selected framing. Right: a continuation sampled at temperature 0.7.

Knowledge that waits for the right document

Averaged over the four facts (geometrically), the right word has probability 0.0348 when the prompt is the start of a statement, 0.0014 after a bare question, and 0.0703 when the question is followed by the start of the answer sentence. The bare question is the worst framing: for the fact about plants, “dioxide” ranks behind 2,266 other tokens after “Answer:”, but is the first choice once the answer sentence has begun. Under the third framing the right word is first or second in 2 of the four facts.

Sampled continuations show the same thing from the other side. After “Answer:” micro writes something in the style of the course — a clause about the heart, then often an end-of-text token and the start of a new, unrelated document — because that is what such text looked like in training. Large base models behave the same way, only more fluently: they will answer a question if documents with that question usually contain its answer, and otherwise continue with another question, a forum reply or a list. Few-shot prompting (plate XXI.3) makes the answer the likely continuation; instruction tuning (Vol. XVII) teaches the model to answer directly.

Base models remain useful precisely because they are not tuned: they are the starting point for every later stage, their probabilities are better calibrated than those of tuned assistants (plate XXII.6), and they are the right tool for scoring or completing text rather than conversing.

Terms

Base model: A model after pretraining only, which continues text as its training documents would.
Framing: How a prompt is written; it decides which kind of document the model thinks it is continuing.
Rank: How many tokens the model considers more likely than a given one.
Instruction tuning: Further training on examples of instructions and good responses, so the model answers directly (Vol. XVII).

A base model continues documents; what it knows comes out when the prompt makes the answer the likely next text.