Vol. IV · Attention Plate 2 of 9query, key, value, attention weights, dot product, soft lookupA

Queries, Keys and Values.

A dictionary returns the value whose key matches exactly. Attention returns a blend of every value, weighted by how well each key matches.

Attention is easiest to understand as a lookup. Every token earlier in the text has been turned into two vectors: a key, which says what it can be found by, and a value, which says what it contributes when found. The current token produces a query, which says what it is looking for. The match between the query and each key is their dot product; a softmax turns the matches into weights that add up to one; and the result is the weighted average of the values.

A Python dictionary would return the one value whose key equals the query and fail otherwise. Attention never fails and is differentiable: it returns a mixture, dominated by the best-matching keys, and gradient descent can adjust the queries, keys and values — which are just linear projections of each token’s vector, Wq x, Wk x, Wv x — until the lookups are useful.

Below: five keys in two dimensions and a query you can turn. The bars show the attention weight each key receives; the output is the weighted average of the five values, 10 to 50. Lengthen the query and the lookup sharpens.

What you will learn

Describe attention as a soft dictionary lookup.
Compute attention weights from dot products and a softmax.
Compute the output as a weighted average of values.
Explain how query length controls how sharp the lookup is.

Workshop

Left: five keys (blue) and the query (red) in two dimensions. Right: the attention weight each key receives. The output is the weights times the values 10, 20, 30, 40 and 50.

A lookup that never fails

With the query pointing at 40°, between keys 1 and 2, the second key gets the largest weight, 55.1%, and the output 16.84 is mostly its value 20 with some of the others mixed in. Lengthen the query to 4 and the dot products spread apart, so the softmax concentrates: the largest weight rises to 69.0%, and a query pointed straight at one key would take nearly all of it. Shorten it to 0.1 and every key gets about a fifth (23.4% at most): the output is close to the plain average, 30. The length of the query — in a trained model, the size of the learned projection — sets how decisive each lookup is.

In a Transformer this happens for every token at once: each position’s query is compared with the keys of all positions it may see (plate IV.4), producing a weight matrix, and each position’s output is its own blend of values. The projections Wq, Wk and Wv are learned, so the model decides what a token advertises in its key, what it looks for in its query and what it hands over in its value. Plate IV.3 adds the one scale factor that keeps the softmax from saturating in high dimensions.

The query–key–value vocabulary came from information retrieval and was made standard by the 2017 Transformer paper; Bahdanau’s 2014 attention computed the match with a small network instead of a dot product.

Terms

Query: The vector a token uses to ask what it is looking for: q = W_q x.
Key: The vector a token is found by: k = W_k x.
Value: The vector a token contributes when attended to: v = W_v x.
Attention weights: The softmax of query–key dot products; non-negative and summing to one.

Attention is a soft lookup: compare a query with every key, softmax the scores, and average the values with those weights.