Encoder-Only Models (BERT)

BERT (Devlin et al., 2018) never generates a word after another word. Every one of its attention layers looks left and right at once, at the full sentence, in every direction, simultaneously. That single design decision, bidirectional attention with no causal mask, is what makes BERT the strongest encoder of a fixed piece of text and, for exactly the same reason, unusable as an open-ended generator without substantial rework.

Masked language modelling

BERT's pretraining objective is not next-token prediction. It is masked language modelling (MLM): randomly select about 15% of input tokens, replace most of them with a [MASK] placeholder, and train the model to predict the original token at each masked position using the entire surrounding sentence, tokens before and after. Because every layer's self-attention is unmasked, a prediction at position 5 can draw directly on position 9's representation, something a causal decoder (see decoder-only-architecture) structurally cannot do at pretraining time.

To reduce the mismatch between training (which sees [MASK] tokens) and downstream fine-tuning or inference (which never does), BERT uses an 80/10/10 corruption recipe on the selected 15%: 80% become [MASK], 10% are replaced with a random token, and 10% are left unchanged but still scored. This forces the model to build genuinely contextual representations rather than a lookup keyed on the literal presence of [MASK].

[CLS], NSP, and what got kept

BERT prepends a special [CLS] token to every input; its final-layer representation is trained (via a lightweight classification head) to summarise the whole sequence, which is what downstream fine-tuning pools for sentence- and pair-level classification. The original paper also trained a next-sentence-prediction (NSP) objective, predicting whether two segments were truly consecutive in the source text. Later work, notably RoBERTa (Liu et al., 2019), found NSP contributed little to downstream quality and dropped it in favour of more data and longer training, one of several findings that BERT was meaningfully under-trained in its original release.

Why bidirectionality is a superpower for understanding

A token's BERT representation is conditioned on the entire input at once, not just what came before it. For tasks where the goal is to understand a fixed span of text, classification, entailment, extractive QA, and especially producing a single fixed-size embedding for retrieval, this is a direct advantage over a causal model, whose representation of an early token can never see later tokens. Sentence-BERT (Reimers and Gurevych, 2019) turned this property into the standard recipe for producing dense sentence embeddings efficiently, and encoder-only backbones remain the dominant choice for embedding models and retrieval encoders (see embeddings-semantic-search) even though generation moved almost entirely to decoder-only architectures.

Why bidirectionality is the wrong tool for generation

Generating text one token at a time requires that, at the moment you predict token t, the model has not yet seen tokens t+1, t+2, ... because they do not exist yet. BERT's attention was never trained under that constraint; every position was trained to condition on the whole sequence. There is no natural way to sample from a BERT-style model token by token in left-to-right order, because doing so would require it to operate in a regime, partial-sequence, fully-bidirectional-within-what-exists, that it never saw during training. This is the structural reason encoder-only architectures were sidelined for open-ended generation, not a matter of scale or tuning.

When it falls down

• No native generation path. Producing open-ended text from an encoder-only model requires either bolting on a decoder (turning it into an encoder-decoder, see encoder-decoder-models-t5) or an iterative masked-refinement scheme, both of which are slower and less natural than a decoder-only model's single autoregressive loop.
• MLM is a low-density training signal. Only the masked ~15% of tokens contribute to the loss on any given forward pass; every non-masked token is "wasted" for that step's gradient, in contrast to a causal model, which gets a next-token loss at every single position on every forward pass (see next-token-prediction-cross-entropy).
• The [MASK] artifact never appears at inference. Fine-tuning and downstream use never present the literal [MASK] token, so the model spends pretraining capacity on an input distribution it will not see in deployment; the 80/10/10 recipe mitigates but does not eliminate this mismatch.
• In-context learning is far weaker. Because BERT was never trained to condition generation on a growing prompt the way decoder-only models were, it does not exhibit the few-shot in-context learning behaviour that made decoder-only models the default interface for general-purpose use (see in-context-learning).

Further reading

• Devlin et al., 2018, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805 - the original architecture and MLM/NSP objectives.
• Liu et al., 2019, RoBERTa: A Robustly Optimized BERT Pretraining Approach, arXiv:1907.11692 - drops NSP, shows BERT was under-trained.
• Reimers and Gurevych, 2019, Sentence-BERT, arXiv:1908.10084 - turns BERT into an efficient embedding model.