Vol. XXII · Decoding Plate 4 of 8repetition penalty, frequency penalty, presence penalty, degeneration, loops, n-gram blockingAE

Repetition and Degeneration.

Penalise what has been said, and the loops stop — along with some of what should have been said again.

Plate XXII.1 showed probability-maximising decoding falling into loops. Sampling makes loops rarer but not impossible, especially at low temperature. The direct remedy is a repetition penalty (Keskar et al., 2019): before sampling, divide the positive logits of every token already present by a factor r > 1 (and multiply negative ones). API services expose variants — a presence penalty subtracted once for any token that has appeared, a frequency penalty proportional to how often — and beam search systems use n-gram blocking, which forbids repeating any n-gram outright.

Every variant acts on tokens, not on meaning, so it cannot tell a degenerate loop from a necessary repeat: the subject of the sentence, a variable name in code, a technical term defined once and used throughout. The course model micro (1.8 million parameters, four layers, trained on this course’s own plates in English and Chinese; plate I.9) runs in the page. The workshop samples at low temperature with the chosen penalty and measures repetition and how probable micro finds the result.

Below: a sample at temperature 0.3 with the chosen repetition penalty; readouts give the share of distinct two-token pairs and the mean log-probability per token under micro, averaged over six samples.

What you will learn
Workshop
micro, prompt “The attention”, temperature 0.3, six samples of 30 tokens; repetition penalty r applied to every token already in the prompt or output (r = 1 is none). Distinct 2-grams and log-probability per token under micro, untruncated, averaged.

Fewer loops, bent probabilities

At temperature 0.3 without a penalty, 72% of the two-token pairs in micro’s samples are distinct and they average −1.75 nats per token. A penalty of 1.3 raises the distinct share to 98%; a penalty of 2 to 100%, but the samples now average −2.59 nats per token: the penalty has pushed the model off its preferred words wherever they had already appeared — including the common words every sentence needs.

The lesson is the general one for decoding hacks: a rule that acts on surface tokens fixes a symptom and costs elsewhere. Better models repeat less in the first place, and the training fix — penalising repetition in the loss, as unlikelihood training does — changes the distribution rather than bending it at inference. In practice penalties are kept mild, applied to a recent window rather than the whole text, and left off for code, where repeating identifiers is correct.

Presence and frequency penalties are additive on logits and so have the same effect on every token regardless of its logit’s sign, unlike the multiplicative penalty above, which behaves differently for positive and negative logits.

Terms

Repetition penalty
Dividing the logits of already-used tokens by a factor greater than one.
Presence penalty
Subtracting a constant from the logit of any token that has appeared.
Frequency penalty
Subtracting an amount proportional to how often a token has appeared.
n-gram blocking
Forbidding any n-gram from appearing twice.
Repetition penalties break loops by acting on tokens, so they also penalise the repetition language needs; keep them mild.

Sources

  1. Keskar et al., “CTRL: A Conditional Transformer Language Model for Controllable Generation”, arXiv:1909.05858, 2019 (repetition penalty).
  2. Paulus, Xiong and Socher, “A Deep Reinforced Model for Abstractive Summarization”, ICLR 2018 (trigram blocking).
  3. Welleck et al., “Neural Text Generation with Unlikelihood Training”, ICLR 2020.