Queries, Keys, and Values

The attention mechanism formula, softmax(QK^T / sqrt(d_k))V, gets written on whiteboards so often that the letters stop meaning anything. Q, K, and V are not arbitrary labels. They are three separate linear projections of the same token embeddings, and the retrieval metaphor they are named after, query, key, value, like a lookup in a hash table, is the cleanest way to understand what the projections are actually doing and why the architecture needs three of them rather than one.

Three projections of the same input

Every token's embedding x gets multiplied by three separate learned weight matrices:

q = x @ W_Q
k = x @ W_K
v = x @ W_V

W_Q, W_K, and W_V are independent parameters, learned by gradient descent like everything else in the network. Before any attention math happens, every token in the sequence has produced its own query vector, its own key vector, and its own value vector. The query asks "what am I looking for," the key answers "what do I contain, for matching purposes," and the value answers "what do I actually offer, if selected." A token's key and its value describe the same underlying content from two different angles: one built for comparison, one built for payload.

Why the dot product is the match

The attention score between a query at position i and a key at position j is their dot product, q_i . k_j. A dot product is large when two vectors point in a similar direction and large in magnitude; it is the standard measure of alignment in a vector space. So W_Q and W_K are trained jointly to produce vectors that align exactly when the underlying tokens should attend to each other, whatever "should" ends up meaning for that particular head, syntactic agreement, coreference, adjacency, whatever the loss finds useful. Nothing forces q and k to live in an interpretable space; the network is free to use any direction it likes, as long as the geometry produces useful matches.

Why not just one matrix

A natural question: why not skip straight to comparing x_i against x_j, or project with a single shared matrix, instead of learning three? Two reasons. First, using the raw embeddings directly for both matching and content would couple two jobs that benefit from being separate: the features that make two tokens "relevant" to each other are not necessarily the same features you want to copy forward once relevance is established. Splitting Q/K from V lets the model learn a matching geometry and a content representation independently. Second, Q and K genuinely need to be different from each other even though they play a symmetric role in the dot product, because attention is directional: token i attending to token j is a different computation from token j attending to token i. A single shared projection would force the match function to be symmetric (q_i . q_j gives the same score as q_j . q_i), which throws away the ability to model asymmetric relationships, a pronoun attending back to its antecedent is not the same relationship as the antecedent attending forward to the pronoun.

The scale factor is not decoration

Once scores are computed, they are divided by sqrt(d_k) before the softmax. This is not a tuning knob added for stability; it corrects a real statistical effect. If q and k have components that are independent with variance 1, their dot product over d_k dimensions has variance d_k, so raw scores grow with head dimension. Large-magnitude scores push softmax into its saturated regime, where the gradient vanishes (see softmax-logits), and training stalls. Dividing by sqrt(d_k) renormalises the variance back to roughly 1 regardless of dimension, which is why the same scale factor works whether d_k is 64 or 128.

The value vector is what actually moves

After softmax turns the scores into weights summing to 1, the output at position i is a weighted sum of every position's value vector: sum_j weight_ij * v_j. The query and key never appear in the output directly; they only decided the weights. This separation matters for intuition: attention is a content-addressable lookup where Q and K compute the address and V supplies the payload, closer in spirit to a soft dictionary read (see attention-as-soft-dictionary-lookup) than to a filter or a gate.

When it falls down

The projections are opaque. Nothing guarantees W_Q and W_K learn a human-interpretable matching criterion. Some heads clearly specialise (positional offset, syntactic role), but many are diffuse or redundant, and post-hoc interpretation of what a given head's Q/K geometry represents is often speculative.
Dimension is a real budget. d_k is typically much smaller than the model dimension (split across heads, see multi-head-attention-deep), so each head's query and key have limited room to encode a matching criterion. Some relational patterns may simply not fit in the available d_k.
Symmetric content, asymmetric use. Because K and V are computed from the same token, a token cannot easily present a different "face" to being matched against versus being copied forward, beyond what two separate linear maps of the same vector allow. This is a real expressive constraint, not just a metaphor.