Vol. XXXI · History and Projects Plate 3 of 14history, attention, Bahdanau, Transformer, parallelism, BLEU, GPT, BERT, T5DL

Attention and the Transformer.

Recurrence made every step wait for the last. Attention let a whole sequence be computed at once.

Sequence-to-sequence translation (2014) squeezed the source sentence into one vector, and long sentences suffered. Bahdanau, Cho and Bengio let the decoder attend: at each output step it scores every encoder state, takes a softmax and reads a weighted average — the mechanism of Vol. IV, born as an add-on to an RNN.

Vaswani and colleagues (2017) removed the RNN. With self-attention, feed-forward layers and positional encodings, every position of a layer can be computed at once, so training uses the whole GPU. Their big model scored 28.4 BLEU on English–German and 41.8 on English–French, records at the time, after 3.5 days on 8 GPUs. The workshop counts the sequential steps each architecture needs to process a sequence.

Left: milestones from attention to T5. Right: the number of steps that must run one after another to process a sequence of T tokens with L layers, for an RNN and a Transformer.

What you will learn
Workshop
Sequential depth when processing a whole sequence in training (the forward pass of a teacher-forced batch): an RNN layer must update its state token by token; a Transformer layer computes all positions at once. Generation remains token by token in both (Vol. XXII). BLEU and training cost from Vaswani et al. (2017), Table 2.

Parallel in training, sequential only where it must be

With a sequence of 512 tokens and 6 layers, an RNN must take 3,072 steps one after another; a Transformer takes 6, each a large matrix multiplication over every position — exactly what GPUs do well. Self-attention costs grow with the square of the length, but that work is parallel; the sequential chain is what had made RNNs slow to train on large corpora. The base Transformer scored 27.3 BLEU on English–German, and the big one 28.4 after 3.5 days on 8 P100s, less training compute than the systems it beat.

Within two years the architecture was everywhere. GPT (2018) pre-trained a Transformer decoder as a language model and fine-tuned it for tasks; BERT (2018) pre-trained an encoder by masking words, and topped benchmarks across the field; GPT-2 (2019) showed that a larger decoder trained on web text could do tasks zero-shot; T5 (2019) cast every task as text to text. The decoder-only branch won for generation, and this course is about it — but the lesson was general: an architecture that uses hardware well can be scaled, and scale, as the next plate shows, was the rest of the story.

The paper’s title was a joke on a Beatles song; its eight authors were listed in random order, marked as equal contributors.

Terms

Attention (in translation)
The decoder reading a weighted average of encoder states.
Self-attention
Every position attending to every position of the same sequence.
BLEU
An n-gram overlap score of translations against references.
Pre-training
Training on generic text before task fine-tuning.
The Transformer removed recurrence so training runs in parallel, and its scaling-friendly design became the basis of every modern language model.

Sources

  1. Bahdanau, Cho and Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate”, ICLR 2015 (arXiv 2014).
  2. Sutskever, Vinyals and Le, “Sequence to Sequence Learning with Neural Networks”, NeurIPS 2014.
  3. Vaswani et al., “Attention Is All You Need”, NeurIPS 2017.
  4. Radford et al., “Improving Language Understanding by Generative Pre-Training”, OpenAI, 2018.
  5. Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, NAACL 2019.
  6. Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, JMLR 2020.