Decoder-Only and Why It Won

Every frontier LLM shipped today, GPT, Claude, Llama, Gemini, Qwen, uses the same skeleton: one transformer stack, causal self-attention, next-token prediction. This was not the obvious choice in 2018. BERT's bidirectional encoder and T5's encoder-decoder pair both had strong empirical results on the benchmarks of the day. Decoder-only won anyway, and the reasons are architectural, not accidental.

The mechanism: one mask, one stack

A decoder-only block is a standard transformer block (anatomy-of-a-transformer-block) with a single constraint on the attention sublayer: a causal mask sets the attention score from position i to any position j > i to negative infinity before the softmax, so after normalisation those positions receive exactly zero attention weight. Each position can only see itself and earlier positions. There is one stack, one set of weights, no separate encoder, no cross-attention sublayer.

Reason one: pretraining and every downstream task share one objective

Next-token prediction (next-token-prediction-cross-entropy) is not a task-specific objective that needs a matching task-specific head. Because any task can be phrased as "continue this text," classification, translation, question answering, and code completion all become instances of the same generation problem with no architectural change and, past a certain scale, no fine-tuning at all: the model conditions on a prompt as if it were "past context" it has already generated, and continues. This is the direct mechanical basis for in-context-learning: a decoder-only model always operates by conditioning purely on everything to its left, so a prompt full of examples is, to the model, simply more left context to condition the next prediction on.

Reason two: it is the cheapest shape to scale and serve

One stack is simpler to shard across accelerators than an asymmetric encoder-decoder pair with a cross-attention dependency between them. And because attention is strictly causal, a token's key and value vectors, once computed, never need to be recomputed as more tokens are generated: the model can cache them and reuse them (kv-cache), so generating token N+1 costs one incremental step, not a full recomputation over all N tokens. BERT-style bidirectional attention cannot make this optimisation at all, since adding a new token could in principle change every existing token's representation.

Reason three: it scaled cleanly, empirically

Radford et al., 2019 (GPT-2) demonstrated that a purely decoder-only model trained on nothing but next-token prediction at sufficient scale developed broad zero-shot task competence, without any task-specific supervision at all. Brown et al., 2020 (GPT-3) pushed this further and showed the same architecture, scaled further, exhibited strong few-shot in-context learning purely from prompt examples. Both results were specifically about what happens when you scale the causal, single-objective recipe; they were not demonstrated for BERT-style or T5-style training at comparable investment, and the field's scaling effort concentrated overwhelmingly on the architecture that had shown this behaviour first.

What it deliberately gives up

None of this makes decoder-only strictly better; it trades away something real. Because attention is causal, an early token's representation can never incorporate information from later tokens, even within the same prompt. A BERT-style encoder gets full bidirectional context on a fixed input essentially for free; a plain decoder-only model does not, which is part of why decoder-only models make weaker off-the-shelf embedding models than dedicated encoders (see encoder-only-models-bert) unless bidirectional attention is deliberately reintroduced over some portion of the input, the idea behind prefix language models.

When it falls down

• Wasted context for pure-understanding tasks. Encoding a fixed document to classify or embed it does not need generation at all, so paying for causal masking, which throws away half the attainable context per token, is a real inefficiency relative to a bidirectional encoder built for exactly that job.
• Left-to-right bias is a genuine information bottleneck, not just a training-time quirk: a decoder-only model's representation of token 3 is permanently blind to tokens 4 through N, which matters for tasks like fill-in-the-middle completion or dense whole-document embedding.
• Dominance reflects investment as well as architecture. Once decoder-only demonstrated strong scaling behaviour, essentially all frontier training compute followed it; that does not prove encoder-only and encoder-decoder are worse in any absolute sense for the workloads they specialise in, only that they stopped receiving comparable scaling investment.
• The single-objective story hides real engineering variation underneath it. "Decoder-only" describes the attention mask, not the rest of the block; sublayer ordering, normalisation placement, and depth-width ratio still vary widely across decoder-only models and materially affect their behaviour (see sublayer-ordering-design and depth-width-aspect-ratio).

Further reading

• Radford et al., 2019, Language Models are Unsupervised Multitask Learners (GPT-2) - the scale-and-unify demonstration.
• Brown et al., 2020, Language Models are Few-Shot Learners (GPT-3), arXiv:2005.14165 - in-context learning emerging from the same recipe at greater scale.
• Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762 - the causal-masked decoder as originally specified, alongside the encoder it was paired with.