Decoder-Only, Encoder-Decoder.
The same blocks with different masks give three kinds of model. The one that only looks backwards turned out to be the one that scales.
The 2017 Transformer had two halves. The encoder read the source sentence with full, bidirectional attention — every word sees every other — and the decoder wrote the translation one token at a time with causal attention, plus cross-attention to the encoder’s output. Within two years the halves had been used separately. BERT (2018) kept only the encoder, trained to fill in masked words: excellent for understanding a text, unable to generate. GPT (2018) kept only the decoder, trained on next-token prediction. T5 (2019) kept both and cast every task as text-to-text.
The difference between them is nothing but the attention mask. Below, the same nine positions under each family’s mask. Decoder-only models won the race to scale for generation for simple reasons: one objective, every token a training signal, the same model for every task through prompting, and a key–value cache that makes generation cheap. Encoders remain in use for search and classification, where reading in both directions helps.
Below: the attention mask of each family over nine positions — a five-token input and a four-token output. A filled cell means the row’s position may attend to the column’s. The table lists a well-known model of each family with its published size.
- Distinguish encoder-only, decoder-only and encoder-decoder Transformers.
- Draw each family’s attention mask.
- Say which objective each family is trained with.
- Explain why decoder-only models became the default for generation.
Three masks, three families
Over nine positions the encoder’s mask allows all 81 pairs, the decoder’s 45 — only the lower triangle — and the encoder-decoder’s 55: full attention inside the input, causal attention inside the output, and every output position reading the whole input through cross-attention. An encoder is trained by hiding words and asking for them back, so every position learns from both sides; it produces excellent representations of a text but has no natural way to write one. A decoder learns from every token of every text by predicting it from the left, and the same act — predict the next token — is how it generates.
By 2020 the largest models were decoders: GPT-3, published that year, had 1,591 times the parameters of BERT-base. The reasons were practical as much as principled: a decoder is one stack with one objective; any task can be posed as text to be continued; training uses every position; and its causal structure allows the cache of keys and values that makes generation fast (Vol. XXIII). Encoder-decoders keep an edge when an input is read once and an output written many times, as in translation, and encoders remain the workhorse of retrieval (plate XXV.3). This course follows the decoder.
Terms
- Encoder-only
- A Transformer with bidirectional attention, trained to reconstruct masked tokens; BERT.
- Decoder-only
- A Transformer with causal attention, trained on next-token prediction; GPT.
- Encoder-decoder
- An encoder for the input and a causal decoder for the output, joined by cross-attention; T5.
- Cross-attention
- Attention whose queries come from the decoder and whose keys and values come from the encoder.
Sources
- BERT: Devlin et al., NAACL 2019 (arXiv 2018).
- T5: Raffel et al., JMLR 2020 (arXiv 2019).
- GPT-3: Brown et al., NeurIPS 2020, Table 2.1.