Drafting Without a Draft Model.
Guess from your own early layers, from extra heads, or from the prompt itself.
Keeping a second model in memory, with the same tokeniser and trained to agree with the first, is a burden. Several methods draft from the target itself. Self-speculation by early exit (LayerSkip, Elhoushi et al., 2024) runs only the first k layers, reads a prediction from there with the final norm and head, and verifies with the full model — cheap, but only accurate if the model was trained so that early layers predict well. Extra heads — Medusa (Cai et al., 2024), EAGLE (Li et al., 2024) — attach small trained modules that predict several tokens ahead from the target’s last hidden state.
The cheapest draft of all is prompt lookup: if the last two tokens appeared earlier in the context, propose the tokens that followed them there. It costs nothing to run and is right exactly when the output copies the input — editing code, quoting a document, summarising with the source’s phrases, or a model stuck in a loop. The course model micro (1.8 million parameters, four layers; plate I.9) is quantised, pruned and sped up in the page, and its loss is measured on held-out course text it never trained on. The workshop decodes 40 greedy tokens with each draft after two prompts: a fresh one and one that repeats itself.
Below: acceptance rate and tokens per target pass for each kind of draft after the chosen prompt; readouts give the chosen draft’s values and estimated speed-up.
- Describe early-exit self-speculation.
- Describe extra-head drafting.
- Explain prompt lookup decoding.
- Match each draft to where it works.
Every draft has a home
micro’s early layers are poor drafters: exiting after one layer, only 12% of draft tokens are accepted after the fresh prompt, and after three layers 20%. micro was never trained to predict from its middle — the logit lens of plate XXVIII.1 shows how unready intermediate layers are — which is why LayerSkip trains with an early-exit loss and layer dropout before it speculates. On repeating text the same three-layer draft is accepted more often (53%), but at three-quarters of a pass per draft token its estimated speed-up is 0.50 — slower than not speculating.
Prompt lookup costs nothing and is useless on fresh text (1.11 tokens per pass), but where the output copies its context it is hard to beat: 1.54 tokens per pass, with 74% of its proposals accepted, for an estimated speed-up of 1.54. Extra-head methods such as EAGLE sit between: a small trained module that drafts well on any text at little cost, now built into the main open-source servers. Lookahead decoding goes without any draft, guessing n-grams by fixed-point (Jacobi) iteration and verifying them in parallel.
Terms
- Self-speculation
- Drafting with part of the target model itself.
- Early exit
- Predicting from an intermediate layer through the final norm and head.
- Medusa / EAGLE heads
- Small trained modules that draft several tokens from the target’s hidden state.
- Prompt lookup
- Proposing tokens that followed the same n-gram earlier in the context.
Sources
- Elhoushi et al., “LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding”, ACL 2024.
- Cai et al., “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads”, ICML 2024.
- Li, Wei, Zhang and Zhang, “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty”, ICML 2024.
- Saxena, “Prompt Lookup Decoding”, 2023 (open-source method).
- Fu, Bailis, Stoica and Zhang, “Break the Sequential Dependency of LLM Inference Using Lookahead Decoding”, ICML 2024.