Encoder-Decoder Models (T5)

T5 (Raffel et al., 2019) made a simple bet: cast every NLP task, translation, summarisation, classification, regression, as text in, text out. Feed the model "summarize: <article>" and train it to emit the summary as free text; feed it "cola sentence: <sentence>" and train it to emit "acceptable" or "unacceptable". One architecture, one loss, one output format for a whole benchmark suite. The architecture that carries this framing is the original transformer's encoder-decoder pair, and understanding it means understanding the one component pure decoder-only models don't have: cross-attention.

Two stacks, three sublayers on one side

An encoder-decoder model runs two separate transformer stacks with independent weights. The encoder reads the entire input with bidirectional attention, exactly like BERT, and produces one contextual vector per input token. The decoder generates the output autoregressively with causal self-attention, exactly like a decoder-only model, but each decoder block has a third sublayer inserted between self-attention and the FFN: cross-attention, where the queries come from the decoder's own current state but the keys and values come from the encoder's final output. Concretely, a decoder block is:

x = x + SelfAttention(LN(x))          # causal, decoder attends to itself
x = x + CrossAttention(LN(x), enc_out) # decoder queries, encoder keys/values
x = x + FFN(LN(x))

This is the mechanism by which the decoder "reads" the input: at every generation step, it can attend directly to any position in the already-fully-computed encoder representation, not just to a compressed summary of it.

Span corruption, not token masking

T5's pretraining objective is a variant of denoising rather than BERT's per-token MLM. Contiguous spans of the input (not individual scattered tokens) are replaced with a single sentinel token, and the target the decoder must produce is just the concatenation of the missing spans, each prefixed by its sentinel. This is more compute-efficient than BERT's approach: the decoder only spends compute generating the corrupted spans, not reconstructing the entire input, so a training example with a short corrupted span costs proportionally less to train on than one where every position must be scored.

T5 also uses a learned relative position bias added to attention scores rather than adding a positional vector to the input embeddings (see positional-encodings), an early example of the relative-position family that RoPE and ALiBi later built on more efficiently.

When two stacks earn their cost

Encoder-decoder is not free: because encoder and decoder have separate weights, an encoder-decoder model of a given "depth" carries roughly twice the parameters of a decoder-only model with the same per-stack depth, for extra cross-attention machinery and a fully separate encoder stack. What that cost buys is a real efficiency win at inference for a specific shape of task: input compressed once, output short and conditioned. The encoder runs exactly once, in full parallel, over the whole input, regardless of how long the generated output ends up being. A decoder-only model, by contrast, must include the entire input inside its single causal sequence and grow its KV cache (see kv-cache) over every input token as well as every generated token. For translation and summarisation, long input, comparatively short output, this asymmetry makes encoder-decoder genuinely cheaper per output token in some regimes, which is why T5-family and later encoder-decoder models remained competitive there long after decoder-only became dominant elsewhere.

When it falls down

Duplicated weights for a narrowing advantage. As decoder-only models scaled and their in-context conditioning got strong enough to handle translation and summarisation with only modest quality loss, the case for paying double the parameters for a dedicated encoder narrowed to specific high-volume, input-heavy, output-light production workloads.
Still autoregressive on the output side. Cross-attention only fixes how the model reads the input; generation is still one token at a time through the decoder, with all the latency and KV-cache costs that implies, no different from decoder-only in that respect.
Fine-tuning-first, not prompting-first. T5's text-to-text framing needed a labelled fine-tuning dataset per task to reach strong performance at its scale; it does not exhibit the strong zero/few-shot in-context learning that made decoder-only models the default general-purpose interface (see in-context-learning).
A fixed encoder representation cannot be revised. Once the encoder has produced its output, it is frozen for the rest of decoding; if useful information about how to interpret the input only becomes clear partway through generation, the encoder has no way to reconsider its representation, whereas a single shared causal stack keeps updating the same shared context as it generates.

Two stacks, three sublayers on one side

Span corruption, not token masking

When two stacks earn their cost

When it falls down

Further reading