Embedding Geometry and Anisotropy

Pick two random, unrelated words and compute the cosine similarity between their contextual embeddings. Intuition says it should land near zero, two random directions in a high-dimensional space are nearly orthogonal by default. Ethayarajh, 2019 measured this directly across BERT, ELMo, and GPT-2, and found it is not near zero at all. Embeddings from these models occupy a narrow cone rather than being spread out isotropically (uniformly in every direction), so even unrelated word pairs show unexpectedly high cosine similarity. This property is called anisotropy, and it is one of the more consequential, least intuitive facts about learned representation spaces.

What causes the cone

A few pressures push in this direction simultaneously. Token frequency is a strong one: very frequent tokens (function words, common subwords) tend to cluster in a distinct region shaped by how often they co-occur with almost everything, and rare tokens sit elsewhere, so the space is organised partly around frequency rather than purely around meaning. Training dynamics play a role too, the softmax and cross-entropy objective that shapes these representations (see softmax-logits, next-token-prediction-cross-entropy) rewards directions that help separate the correct next token from a huge vocabulary, and this can incidentally push many representations toward a shared dominant direction that helps normalisation rather than one that reflects meaning. The effect is not uniform across depth: it tends to worsen in upper layers, where representations become more specialised to the immediate context, which is one reason anisotropy interacts with the contextual-versus-static distinction (see static-vs-contextual-embeddings).

Why it matters in practice

If most pairs of embeddings already have high cosine similarity by default, then a similarity score of 0.75 between two supposedly related items is far less informative than it looks, the baseline for "unrelated" might already be 0.6 or higher in that space. Any system that does nearest-neighbour search or similarity ranking on raw embeddings (see cosine-similarity-vs-dot-product, sentence-embeddings-and-pooling) inherits this compression: scores get squeezed into a narrow high range, and the genuinely discriminative signal is a small fraction of the total similarity value.

Two mitigation strategies emerged. One is a post-hoc geometric fix: standardise or "whiten" the embedding distribution (subtract the mean, decorrelate dimensions) before comparing vectors, an approach explored under names like BERT-flow and BERT-whitening. The other is a training-time fix: explicitly optimise for a more isotropic, discriminative space using a contrastive objective, the approach taken by SimCSE (Gao et al., 2021), which pulls semantically similar sentence pairs together and pushes unrelated ones apart during fine-tuning rather than relying on whatever geometry pretraining happened to produce.

When it falls down

• Naive "just take cosine similarity" pipelines silently inherit this bias. Without correction, everything looks moderately similar to everything else, and genuine semantic distinctions get compressed into a small usable range near the top of the scale.
• The degree of anisotropy varies by model, checkpoint, and layer. A whitening transform or similarity threshold tuned for one model's geometry does not transfer cleanly to another, or even to a different layer of the same model.
• Anisotropy is not unconditionally bad. Some of the shared directions encode genuinely useful information, such as frequency, and aggressively "fixing" the geometry can discard signal a downstream task actually needed.
• Measurement methodology differs across papers. Whether the mean vector is subtracted before measuring, which layer is examined, and which corpus is used all change the reported anisotropy numbers, so figures are not always directly comparable study to study.