Perplexity - Measuring a Language Model

Before there were benchmarks with names, there was perplexity. It is the intrinsic measure of how well a language model predicts text, it falls directly out of the training loss, and despite being decades old it is still the number researchers watch during pretraining. It is also the metric people misuse most often, because it looks comparable across models when it usually is not.

What it is

Perplexity is the exponential of the average per-token cross-entropy loss:

perplexity = exp( - (1/N) * sum_t log P(x_t | x_<t) )

Because it is just exp(loss), minimising loss and minimising perplexity are the same objective (see next-token-prediction-cross-entropy). The exponential puts it on an interpretable scale. Perplexity P means the model is, on average, as uncertain as if it were choosing uniformly among P equally likely tokens at each step. A perplexity of 1 is perfect prediction (the model always knew the next token); a perplexity equal to the vocabulary size is a model that learned nothing. Real LLMs on English text land somewhere in the low tens.

A concrete reading

Suppose a model assigns the actual next token an average probability of 1/12. Its perplexity is 12: it is as confused as someone rolling a fair 12-sided die at each token. Halve the perplexity to 6 and the model is now, on average, twice as confident in the right token. This is why perplexity improvements feel small numerically but matter: dropping from 12 to 11 means measurably better prediction across every token in the corpus.

The comparison trap

Perplexity is per-token, and tokens are not universal. Model A with a 32k-vocabulary tokeniser and model B with a 128k one chop the same sentence into different numbers of pieces, so their per-token losses are not measuring the same thing (see tokenisation-bpe). A model with a coarser tokeniser predicts fewer, larger tokens and can post a lower perplexity while being no better at modelling the language. The only valid comparisons are: same tokeniser, same evaluation text, same context handling. Cross-tokeniser claims of "lower perplexity" are close to meaningless, and normalising to bits-per-byte is the usual fix when you must compare across tokenisers.

When it falls down

• It measures fit, not usefulness. A model can have excellent perplexity on web text and still be unhelpful, unsafe, or bad at reasoning. Perplexity rewards predicting the corpus, including its errors and boilerplate; it is blind to whether the output is what a user wanted (which is what instruction tuning and evals address).
• Domain dependence. Perplexity is only defined relative to a test distribution. A model has different perplexity on code, on legal text, and on casual chat; a single number without a stated corpus is uninterpretable.
• It rewards hedging. Because it is an average log-probability, a model that spreads probability to stay non-committal on hard tokens can score better than one that commits and is occasionally wrong, even when commitment is what the task needs.

Further reading

• Jurafsky and Martin, Speech and Language Processing, Ch. 3 (N-gram language models and perplexity) - the clearest textbook treatment.
• Chip Huyen, Evaluation Metrics for Language Modeling - perplexity, entropy, and bits-per-character in one place.