Self-Attention vs Cross-Attention

The difference between self-attention and cross-attention is a single sentence: in self-attention, Q, K, and V all come from the same sequence; in cross-attention, Q comes from one sequence and K/V come from another. That is the entire mechanical distinction. What it implies architecturally, and why most current large language models have quietly dropped cross-attention altogether, is worth spelling out.

Same weights, different wiring

Both self-attention and cross-attention use the identical formula, softmax(QK^T / sqrt(d_k))V, and the identical projection machinery from queries-keys-values. Nothing about the operation itself changes. What changes is which tensor gets projected into Q and which tensor gets projected into K and V.

# self-attention: everything from x
Q = x @ W_Q ; K = x @ W_K ; V = x @ W_V

# cross-attention: queries from decoder state y, keys/values from encoder output x
Q = y @ W_Q ; K = x @ W_K ; V = x @ W_V

In self-attention, every position can attend to every other position (subject to masking, see causal-masking) within the same sequence, so the sequence is, in effect, building context out of itself. In cross-attention, the querying sequence never attends to itself in that operation; it looks outward at a second, fixed sequence and asks which of its elements are relevant right now.

Where cross-attention lived: the encoder-decoder bridge

The original transformer (Vaswani et al., 2017) was a machine-translation architecture with an encoder and a decoder. The encoder ran self-attention over the source sentence to build a contextualised representation. The decoder ran self-attention over the target sentence being generated, and, at each decoder layer, a cross-attention step where the decoder's queries attended to the encoder's keys and values. This is the literal bridge between the two stacks: it is how "the word being generated right now" gets to look back at "the source sentence that needs translating." T5 and the original BART kept this structure faithfully. It is also exactly how encoder-decoder models condition generation on an external input like an image caption feature map or an audio encoding, seen in models such as Whisper, where the decoder's cross-attention keys and values come from the audio encoder.

Why decoder-only LLMs mostly dropped it

GPT-style decoder-only models, which is most of the current generation of large language models, have no separate encoder stack, so there is no second sequence to cross-attend to. Instead, the prompt and the generated continuation sit in one sequence, and self-attention (causally masked) does the entire job: a token generated late in the sequence can attend back to the prompt the same way it attends to earlier generated tokens, because they are all just prior positions in the same stream. This architectural simplification, one stack, one attention type, one set of weights reused for both "reading" the prompt and "generating" the continuation, is a large part of why decoder-only became the default: it removed an entire component, and it turned out not to cost the model anything decoder-only training could not recover through scale.

Cross-attention is not gone, just relocated

Cross-attention persists wherever a model needs to condition generation on a genuinely separate modality or a fixed external representation that is expensive to recompute per token. Multimodal architectures often use cross-attention to let a text decoder attend to a frozen image or audio encoder's output (rather than concatenating image patch tokens directly into the text sequence and paying full self-attention cost over them). Retrieval-augmented architectures have also used cross-attention to let a decoder attend to retrieved passage representations without embedding the entire retrieval corpus into the token sequence.

When it falls down

• Cross-attention needs two well-defined sequences. It presumes a clean querying side and a clean context side. Tasks where the boundary is fuzzy, where the "context" should itself be updated by what the query attends to, fit self-attention's single unified sequence better.
• It cannot see itself. A pure cross-attention layer gives the decoder no way to relate two of its own generated tokens to each other; that is still self-attention's job. Encoder-decoder blocks always pair the two, never substitute one for the other.
• The "decoder-only wins" story oversimplifies. Encoder-decoder models remain competitive for tasks with a clear, fixed, one-shot input such as translation, and can be more compute-efficient there because the encoder computes the source representation once rather than re-deriving it inside every causal step.
• Multimodal cross-attention adds engineering surface. It requires the two modalities to be projected into a compatible key/value space and typically forces the base language model's weights to accommodate a new sublayer, which is one reason some multimodal designs favour concatenating modality tokens into a single self-attention stream instead.

Further reading

• Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762 - defines the encoder-decoder cross-attention bridge.
• Raffel et al., 2019, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5), arXiv:1910.10683 - a widely studied encoder-decoder model built on this split.