Vol. XXV · Tools and Agents Plate 3 of 9retrieval, retrieval-augmented generation, RAG, BM25, TF-IDF, embeddings, dense retrieval, recallAE

Retrieval.

Find the right page, put it in the context, then answer from it.

A model knows only what its weights absorbed before its cutoff, and recalls even that imperfectly (plate XII.9). Retrieval-augmented generation searches a document collection for passages relevant to the question and places them in the context, so the answer can be read rather than remembered — and cited. The search is the hard part. Lexical methods score passages by shared words: TF-IDF weighs rare words more, and BM25 (Robertson and colleagues, 1990s) adds saturation of repeated words and normalisation by length. Dense methods embed query and passage as vectors and compare them, which can match meaning across different words — if the embedding was trained to.

The workshop indexes 300 passages from the course’s held-out plates. Each of 60 queries is a few content words drawn from one passage, optionally mixed with words from other passages as noise; a query succeeds at k if its source passage is among the top k results. It compares BM25, TF-IDF and the mean of micro’s token embeddings.

Below: recall at k — the share of queries whose source passage is in the top k — for the three methods, at the chosen query length and noise.

What you will learn
Workshop
300 passages (paragraphs of at least 200 characters) from the held-out English plates; 60 queries, each the chosen number of content words drawn from one passage plus the chosen number of words from other passages. BM25 with k₁ = 1.2, b = 0.75; TF-IDF cosine; micro’s token embeddings averaged and compared by cosine. Recall at k over the 60 queries.

Words match; embeddings must learn to

With two words from the passage and no noise, BM25 ranks the source first for 82% of queries and TF-IDF for 65%; add six words from other passages and BM25 still finds it first 45% of the time — top five 90% — while TF-IDF falls to 15%. BM25’s saturation and length normalisation keep one repeated or common word from swamping the rest. micro’s averaged token embeddings rank the source first for only 2% of queries at the chosen setting and in the top ten for 7%: a language model’s embeddings encode what tokens do, not what passages are about.

Dense retrievers are therefore trained for the job — DPR (2020) trained encoders contrastively on question–passage pairs, pushing each question towards its passage and away from others — and they win where queries and documents use different words; BM25 wins on exact names, codes and rare terms. Production systems usually combine both and rerank the candidates with a model that reads query and passage together. Retrieval also changes what the model must do: answer from the passages, cite them, and say so when they do not contain the answer.

The queries here are made of the passages’ own words, which flatters lexical search; real questions paraphrase, which is where trained embeddings earn their place.

Terms

Retrieval-augmented generation
Searching documents and placing results in the context before answering.
BM25
A lexical ranking function with term saturation and length normalisation.
Dense retrieval
Ranking by similarity of learned query and passage embeddings.
Recall at k
The share of queries whose relevant passage is in the top k.
Retrieval puts evidence in the context; lexical search matches words, and embeddings match meaning only when trained to.

Sources

  1. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS 2020.
  2. Robertson and Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond”, Foundations and Trends in Information Retrieval, 2009.
  3. Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering”, EMNLP 2020.