Sentence Embeddings and Pooling

Search, clustering, and deduplication over documents need one vector per passage, but a transformer's native output is one contextual vector per token (see static-vs-contextual-embeddings). Getting from many vectors to one is called pooling, and the pooling method chosen is not a minor implementation detail, it materially changes what the resulting embedding is good and bad at.

The main pooling strategies

CLS-token pooling takes the final vector of a special token prepended to the sequence, a convention from BERT (Devlin et al., 2018), where this token's representation was trained, via a next-sentence-style objective, to summarise the whole input. In practice, the raw, untuned CLS vector is a mediocre sentence embedding; it was trained for a specific auxiliary task, not general-purpose similarity.

Mean pooling averages every token's final vector (with padding tokens masked out of the average). It is simple, requires no special token, and is a reasonably strong default, though it treats a content-bearing word and a function word like "the" as equally important contributors to the average.

Max pooling takes, per dimension, the maximum value across all token vectors, capturing the strongest activation for each feature rather than the average. It is used less often than mean pooling for general sentence similarity.

Last-token pooling takes the final token's vector, common in decoder-only, causally-masked models, since the last token is the only position that has attended to the entire preceding sequence. Many modern embedding models built on top of large autoregressive language models use this approach.

Why fine-tuning changes everything

Reimers and Gurevych, 2019 showed that raw BERT embeddings, whether CLS or mean-pooled, perform poorly on semantic similarity benchmarks, in some cases worse than decade-old static embeddings like GloVe. Their fix, Sentence-BERT, fine-tunes the model with a siamese or triplet network structure specifically to make pooled embeddings from related sentences close and unrelated sentences far apart under cosine similarity (see cosine-similarity-vs-dot-product). This also solved a practical problem: comparing every pair among n sentences with a full cross-encoder is O(n^2) expensive forward passes, while precomputing pooled sentence embeddings once and comparing vectors is cheap, the approach that makes large-scale semantic search tractable.

When it falls down

• Mean pooling dilutes meaning with function words. A long sentence's average vector can be pulled toward the generic geometry of "the", "of", and "and", which contribute little semantic content but equal weight.
• CLS pooling without task-specific fine-tuning is unreliable. Despite being architecturally positioned to summarise the sequence, an untuned CLS vector inherits the anisotropy problems common to contextual embeddings (see embedding-space-geometry-anisotropy).
• Pooling method and training objective are coupled, not swappable. A model fine-tuned assuming mean pooling will not necessarily produce good embeddings if you switch to last-token pooling at inference time; the two must match.
• Padding must be masked correctly. Including padding tokens in a mean-pooling average is a common implementation bug that silently degrades embedding quality specifically for batched inference, where sequences of different lengths are padded to match.

Further reading

• Reimers and Gurevych, 2019, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, arXiv:1908.10084 - the fine-tuning approach that made pooled embeddings competitive for similarity tasks.
• Devlin et al., 2018, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805 - origin of the CLS-token pooling convention.