word2vec and GloVe

Before transformers, before attention, there was a simpler question: can a dense vector, trained on nothing but raw text, capture what a word means well enough that vector("king") - vector("man") + vector("woman") lands near vector("queen")? Mikolov et al., 2013 answered yes, and the resulting embeddings, word2vec, became the default way to represent words for years, alongside a second, differently-derived method, GloVe.

word2vec: predict the neighbours

word2vec trains a shallow model on one of two symmetric tasks. Skip-gram predicts the surrounding context words given a target word; continuous bag-of-words (CBOW) predicts the target word given its context. Neither task is the point; the point is the by-product. After training, the hidden-layer weight matrix, one row per vocabulary word, becomes the word's embedding, and it turns out that words used in similar contexts end up with similar vectors, purely because predicting similar contexts well rewards similar representations. This is the distributional hypothesis made operational: meaning is inferred from company kept, not defined explicitly.

Training the full softmax over a large vocabulary at every step is expensive, so the practical version uses negative sampling, turning the task into distinguishing real context words from a handful of random "negative" words, a cheap approximation that made training on billions of words feasible on ordinary hardware at the time.

GloVe: factor the co-occurrence matrix

Pennington et al., 2014 took a different route to a similar destination. GloVe first builds a global word-word co-occurrence count matrix over the entire corpus (how often does word j appear near word i, summed over every occurrence), then learns vectors whose dot products approximate the log of those co-occurrence ratios. Where word2vec never looks past a local sliding window at each training step, GloVe optimises directly against corpus-wide statistics. In practice the two methods produce embeddings of comparable quality and similar geometric properties, evidence that the co-occurrence statistics themselves, not the specific training procedure, are what determines the structure.

Both word2vec and GloVe produce static embeddings: exactly one vector per word type, fixed after training, with no sensitivity to the sentence it appears in (see static-vs-contextual-embeddings). Cosine similarity between these vectors became the standard way to measure word relatedness (see cosine-similarity-vs-dot-product), and the analogy arithmetic that made both famous is a direct consequence of the vectors being trained by a linear objective, linear relationships in meaning tend to show up as linear relationships in the vector space.

When it falls down

One vector per word ignores polysemy. "Bank" gets a single vector that has to represent both the riverbank sense and the financial sense simultaneously, typically landing somewhere between the two, useful for neither in isolation.
Corpus bias is baked directly into the geometry. Analogy tasks trained on real text reliably reproduce gender and other social biases present in the training corpus, because the method has no mechanism to distinguish "statistically associated" from "should be associated".
Fixed vocabulary, no handling of unseen words. A word absent from training has no row at all; subword extensions like fastText were built specifically to patch this gap by composing embeddings from character n-grams.
Analogy arithmetic is a compelling demo, not a rigorous benchmark. The curated examples that made these methods famous work well; systematic evaluation shows the linear-analogy property is inconsistent across relation types and sensitive to preprocessing choices.

word2vec: predict the neighbours

GloVe: factor the co-occurrence matrix

What both share, and what neither has

When it falls down

Further reading