Masked Language Modelling

Ask a causal model to fill in a blank in the middle of a sentence and it cannot, structurally: it has never seen anything to the right of the blank. Masked language modelling (MLM) exists to train exactly that skill. Devlin et al., 2018 introduced it as BERT's pretraining task, and the specific corruption recipe they chose is more deliberate than "hide 15% of the words," with each design choice fixing a real failure mode.

The 80/10/10 recipe

BERT selects roughly 15% of input token positions per sequence, then does one of three things to the selected positions:

for each selected position:
    80% of the time -> replace token with [MASK]
    10% of the time -> replace token with a random vocabulary token
    10% of the time  -> leave the original token unchanged

The model is trained to predict the original token at every selected position, using context from both directions, regardless of which of the three corruptions was applied. Why not simply mask 100% of the selected positions with [MASK]? Because [MASK] is a training-only artefact. If every corrupted position were [MASK], the model could learn a shortcut tuned to that specific token, and at fine-tuning time, when [MASK] never appears in real input, that shortcut is useless. Injecting random tokens 10% of the time forces the model to build genuinely contextual representations for every input token, not just the ones flagged as corrupted, because it can never be sure whether the token in front of it is the real one, a random one, or the answer to a masked position. Leaving 10% unchanged closes the loop: the model must produce a correct prediction even when there is nothing wrong with the input at all.

Why bidirectionality is the whole point

A causal model conditions each prediction on x_<t only, by construction (see causal-lm-pretraining-task). MLM removes that restriction: the encoder has unmasked self-attention, and each masked position is predicted using both left and right context simultaneously. This produces representations that are strictly richer for understanding tasks, since "the trophy didn't fit in the suitcase because it was too big" cannot be resolved by left context alone. That is exactly why encoder models trained with MLM remain the workhorse for embeddings and retrieval (see embeddings-semantic-search) even in a world dominated by decoder-only generative models.

The cost: no native generation

The flip side of bidirectionality is that MLM never trains the model to produce a sequence left to right. At inference, you cannot ask a BERT-style encoder to "continue this text," because it has never predicted a token without also being allowed to see tokens after it. Turning an encoder into a fluent generator means either bolting on a decoder (moving toward span-corruption) or abandoning the pretrained representations for a generation head entirely. This asymmetry, not model quality, is the main reason the generative-assistant era consolidated around causal pretraining (see pretraining-objectives) rather than MLM.

Next sentence prediction, and why it was dropped

BERT's original recipe paired MLM with a second objective, Next Sentence Prediction (NSP): given two spans, predict whether the second genuinely follows the first in the source document or was swapped in at random. The intent was to teach sentence-pair relationships useful for tasks like natural language inference. Liu et al., 2019 (RoBERTa) later showed NSP contributes little to downstream performance and can be dropped in favour of just packing contiguous sentences and training MLM for longer, with better results. RoBERTa's other changes, dynamic masking (recomputing which tokens are masked on every epoch instead of once at preprocessing time) and much larger batches, mattered more than NSP ever did.

When it falls down

• Pretrain/fine-tune mismatch remains, just smaller. Even with the 80/10/10 mix, [MASK] tokens appear during pretraining and essentially never during fine-tuning or inference; the mitigation reduces the mismatch, it does not eliminate it.
• Masking independence is a simplification. The loss treats each masked position as conditionally independent given the corrupted context, but real language has strong dependencies between nearby masked tokens (mask two adjacent words in the same phrase and predicting one should inform the other); MLM's factorisation ignores this.
• Fixed 15% is a tuned constant, not a law. Higher masking rates trade off training signal per token against context available for each prediction; different corpora and model sizes shift the optimum, and work since BERT has revisited the ratio.
• No natural notion of sequence length or stopping. Because MLM never generates a sequence, it has nothing analogous to an end-of-sequence token to learn; that entire vocabulary of generation control is absent from what the objective teaches.

Further reading

• Devlin et al., 2018, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - the original masking recipe and NSP.
• Liu et al., 2019, RoBERTa: A Robustly Optimized BERT Pretraining Approach - drops NSP, adds dynamic masking, and retunes the training recipe.