Span Corruption

BERT reconstructs a prediction at every masked position across a full-length sequence (see masked-language-modelling); a decoder, if the model has one, would have to process a target as long as the input to match. Raffel et al., 2019 (T5) asked what happens if you corrupt contiguous spans instead of scattered single tokens, and only ever generate the missing spans, not the whole document. The answer was an objective that is both a better fit for encoder-decoder architectures and noticeably cheaper to train.

The corruption recipe

T5 samples spans of consecutive tokens from the input (mean span length around 3, roughly 15% of tokens corrupted overall, matching BERT's rate for comparability), and replaces each entire span with a single unique sentinel token rather than one [MASK] per hidden token:

original:   Thank you for inviting me to your party last week .
input:      Thank you <X> me to your party <Y> week .
target:     <X> for inviting <Y> last <Z>

The encoder sees the corrupted input with sentinels standing in for the missing spans. The decoder, conditioned on the encoder's output, autoregressively generates only the concatenation of sentinel-plus-span pairs, in order, terminated by a final sentinel. Every other input token, the 85% that was never corrupted, is never reconstructed at all.

Why the target being short matters

BERT's masked-token loss is computed over every masked position in a sequence of length T, but crucially, BERT has no decoder: it emits one classification per masked position in parallel, so this is cheap. T5's setup is different because it has a genuine autoregressive decoder, and decoding is where cost lives. Generating a target as long as the input, one sentinel and one token at a time, would multiply training cost substantially. By collapsing each span down to a single sentinel and only emitting the missing content, span corruption keeps the decoder's target sequence a small fraction of the encoder's input length, so the expensive autoregressive half of the model does far less work per example than reconstructing the whole document would require.

Bidirectional input, autoregressive output

Span corruption is not a drop-in replacement for either causal LM or MLM; it needs both halves of an encoder-decoder. The encoder attends bidirectionally over the corrupted input, giving it the same "see both directions" advantage as MLM (see masked-language-modelling). The decoder is causally masked over its own generated span tokens, just like a normal autoregressive model (see causal-lm-pretraining-task), but it additionally attends to the full encoder output via cross-attention. This is why T5 reframes every task, translation, summarisation, classification, as text-to-text: the architecture already expects "corrupted input in, missing text out," and downstream tasks are just a different flavour of the same shape.

Span corruption versus fill-in-the-middle

Both objectives train a model to produce text that belongs somewhere other than the end of a sequence. Span corruption does it by giving the model a genuinely bidirectional encoder and a separate decoder; fill-in-the-middle achieves a similar-looking capability with a plain causal decoder, by rearranging training documents rather than changing the architecture. Span corruption is the right choice when you are training (or already committed to) an encoder-decoder; FIM is the right choice when you want infilling behaviour inside a purely causal, decoder-only model.

When it falls down

Sentinel vocabulary is a fixed budget. T5 reserves 100 sentinel tokens; a document with unusually dense corruption or many small spans can exhaust them, forcing coarser span choices.
Span length is a hyperparameter with real consequences. Very short spans (length 1) collapse toward BERT-style masking; very long spans remove so much context that the encoder input becomes uninformative and the decoder is effectively asked to hallucinate long stretches of text.
Needs an encoder-decoder, which costs more parameters for the same active compute than a decoder-only model at inference, because you carry both an encoder and a decoder stack rather than reusing one stack for everything.
Task-to-text reframing is elegant but not free. Coercing classification into "generate the class name as text" discards the direct, well-calibrated softmax-over-classes head that a dedicated classification model would use.

The corruption recipe

Why the target being short matters

Bidirectional input, autoregressive output

Span corruption versus fill-in-the-middle

When it falls down

Further reading