Vol. XXXI · History and Projects Plate 2 of 14history, neural language models, Bengio, RNN, LSTM, word2vec, perplexity, Penn TreebankDL

Neural Language Models.

Represent words as vectors, learn them by predicting, and similar words share what they learn.

Bengio and colleagues (2003) proposed the model behind plate I.7: map each word to a learned vector, feed the previous words’ vectors through a neural network, and output a distribution over the next word. Because similar words get similar vectors, evidence about one generalises to others — the remedy for the sparsity that limited n-grams. It was too expensive to train at scale for a decade.

Recurrent networks removed the fixed window: Mikolov’s RNN language models (2010) carried a hidden state through the whole text, LSTMs (Hochreiter and Schmidhuber, 1997) made that state trainable over long spans, and word2vec (2013) showed that cheap training of word vectors captured semantic relations. The workshop charts the Penn Treebank test perplexity reported by successive methods.

Left: milestones. Right: reported Penn Treebank test perplexities, from a 5-gram model to GPT-2, on a logarithmic scale.

What you will learn
Workshop
Word-level Penn Treebank test perplexities as reported: Kneser–Ney 5-gram and RNN (Mikolov et al., 2011), large regularised LSTM (Zaremba et al., 2014), AWD-LSTM (Merity et al., 2018), GPT-2 1.5B zero-shot (Radford et al., 2019, different tokenisation and detokenised text, so only indicative).

Vectors that share what they learn

On the Penn Treebank — a million words of Wall Street Journal text with a 10 000-word vocabulary, the field’s standard benchmark for a decade — the best counted model scored 141.2 perplexity, a simple RNN 124.7, a regularised LSTM 78.4 in 2014, and a carefully tuned LSTM 57.3 in 2018: a 2.5-fold fall from learned representations, recurrence and regularisation. GPT-2’s zero-shot 35.8 came from a model never trained on the benchmark at all — a sign of what the next plates bring.

Two ideas from this period run through the whole course: embeddings (Vol. III), vectors learned by prediction in which meaning becomes geometry; and the language model as a general pre-training objective, which ELMo (2018) used to give every word a vector that depends on its context. What held recurrent models back was sequential computation — a hidden state must be updated one token at a time, which wastes parallel hardware — and the difficulty of carrying information across hundreds of steps.

word2vec’s famous analogy — king − man + woman ≈ queen — made embeddings vivid, though later analyses showed such analogies are less reliable than the demonstrations suggested.

Terms

Neural language model
A neural network predicting the next word from learned word vectors.
Recurrent network
A network carrying a hidden state from token to token.
LSTM
A recurrent cell with gates that preserve information over long spans.
Word embedding
A learned vector for each word.
Neural language models replaced counts with learned vectors and recurrence, cutting benchmark perplexity several-fold — until sequential computation became the bottleneck.

Sources

  1. Bengio, Ducharme, Vincent and Jauvin, “A Neural Probabilistic Language Model”, JMLR 2003.
  2. Mikolov et al., “Recurrent neural network based language model”, Interspeech 2010.
  3. Hochreiter and Schmidhuber, “Long Short-Term Memory”, Neural Computation, 1997.
  4. Mikolov et al., “Distributed Representations of Words and Phrases and their Compositionality”, NeurIPS 2013.
  5. Zaremba, Sutskever and Vinyals, “Recurrent Neural Network Regularization”, arXiv:1409.2329, 2014.
  6. Merity, Keskar and Socher, “Regularizing and Optimizing LSTM Language Models”, ICLR 2018.