NLP Foundations
Static vs Contextual Embeddings
The shift from one vector per word to one vector per word-in-context, and why that shift is arguably the single biggest reason transformer-based models outperformed earlier NLP.
intermediate · 7 min read
word2vec and GloVe (see word2vec-glove-history) give "bank" exactly one vector, forever, regardless of whether the sentence is about a river or a mortgage. A transformer gives "bank" a starting vector that is identical in both cases, then changes it. By the time that token's representation reaches the final layer, it has been mixed, through repeated attention over every other token in the sequence, into something that looks different depending on what it sat next to. That difference, static versus contextual, is the shift that separates pre-2018 word representations from everything that came after.
What actually changes
The input embedding lookup (see input-embeddings-and-lookup) is still static; row i of the embedding matrix is the same regardless of context. Contextualisation happens afterward, through the transformer stack. Each attention layer lets a token's vector absorb information from other tokens it attends to (see attention-mechanism), and this happens repeatedly across every layer, accumulating in the shared residual stream (see the-residual-stream). "River bank" and "bank account" start with the identical row for "bank" and diverge as they pass through the network, because each occurrence attends to a different neighbourhood of tokens.
ELMo (Peters et al., 2018) was the first widely adopted contextual method, using a bidirectional LSTM to produce context-sensitive vectors before transformers took over the role. BERT (Devlin et al., 2018) then made transformer-based contextual embeddings the default, and essentially every model since has followed the same pattern: static lookup in, contextual representation out.
A gradient, not a switch
Contextualisation is not binary. Early transformer layers tend to preserve much of the token's original identity; later layers are where representations become sharply context-specific, entangled with syntax, coreference, and task-relevant signal. This is why interpretability probes that ask "what does layer k know" get different answers at different depths, and why techniques like the logit lens track how a prediction sharpens across layers rather than existing fully-formed at layer one.
When it falls down
- Contextual embeddings are not portable. A vector for "bank" from one model, one layer, one context is not comparable to a vector from a different model or a different layer; there is no single "the" contextual meaning the way word2vec offered a single stable word vector.
- You cannot precompute a lookup table for contextual embeddings. Static embeddings for a vocabulary can be computed once and reused; contextual embeddings require a forward pass per sentence, which is far more expensive for tasks like large-scale semantic search (see embeddings-semantic-search).
- Early layers can look deceptively "static". Probing an early layer and finding it behaves like a static embedding does not mean the model has no contextual computation planned there, only that it has not been expressed yet.
- Contextual similarity inherits geometric distortions. Comparing contextual vectors with cosine similarity runs into the anisotropy problem, where unrelated words end up with unexpectedly high similarity scores (see embedding-space-geometry-anisotropy).
Further reading
- Peters et al., 2018, Deep Contextualized Word Representations, arXiv:1802.05365 - ELMo, the pre-transformer contextual embedding method.
- Devlin et al., 2018, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805 - the model that made transformer-based contextual embeddings standard.