NLP Foundations
Log-Probs and Token Probabilities
What the log-probability numbers an API actually returns mean, how to turn them back into probabilities and perplexity, and the practical uses (reranking, confidence checks, hallucination flags) that make them worth reading instead of skipping.
beginner · 6 min read
Most LLM APIs will hand you a log-probability for every generated token, and often for the top alternatives the model considered at each position, if you ask for it. Almost nobody asks. The generated text is the visible product; the logprobs are the receipt showing exactly how confident the model was at every single step, and reading that receipt turns out to be useful for more than curiosity.
What the number actually is
A logprob of -0.05 means the model assigned the token it produced a probability of exp(-0.05) ~= 0.95. A logprob of -4.6 means a probability of about 0.01.
probability = exp(logprob)
These are natural logs (base e), matching the same units used internally by the training loss (see next-token-prediction-cross-entropy), not log2. To convert to Shannon bits, divide by ln(2) ~= 0.693 (see entropy-and-surprise).
Sequence probability is a sum in log space
The chain rule says the probability of a whole generated sequence is the product of its per-token conditional probabilities. In log space, that product becomes a sum: add up every returned per-token logprob and you have the log-probability of the entire completion. This is exactly the quantity averaged and negated to compute training loss, and exponentiated to compute perplexity. A completion with a low average per-token logprob is, informally, one the model surprised itself while writing.
Three practical uses
- Reranking. Generate several candidate completions (see sampling-decoding) and rerank them by total or length-normalised sequence logprob, a cheap proxy for "which completion did the model itself find most natural," without any extra model calls.
- Low-confidence flagging. A sharp drop in per-token logprob, a specific token the model assigned surprisingly low probability to, often lines up with a fabricated fact or an invented number. Token-level logprob is a cheap first-pass signal for this, useful as a filter, not reliable enough to trust on its own.
- Direct answer verification. For multiple-choice or classification-style tasks, compare the logprob the model assigns to each candidate label token directly, instead of generating free text and parsing it. It is faster and eliminates an entire class of formatting-parsing failures.
When it falls down
- Sequence logprob is length-biased. Longer completions accumulate more negative log-probability simply from having more terms in the sum, so ranking raw total logprob across completions of different lengths systematically favours shorter ones. Length normalisation (dividing by token count) is the common fix, but it is a heuristic correction, not a principled one, and it changes which completion wins.
- Low logprob correlates with, but does not prove, factual error. A token can be locally surprising for entirely benign reasons, a rare but correct proper noun, an unusual but valid stylistic choice, the first token after an abrupt topic change, so using it as a hallucination detector produces real false positives and real false negatives.
- Post-RLHF logprobs are a weaker confidence signal. Instruction tuning and RLHF change what token probabilities represent (see calibration-of-language-models), so reading "the model's confidence" directly off a chat-tuned model's logprobs is less reliable than the same reading on a base pretrained model.
Further reading
- OpenAI API reference, Chat Completions - the practical shape of a logprobs response object.
- Holtzman et al., 2019, The Curious Case of Neural Text Degeneration, arXiv:1904.09751 - analyses what token-level probability looks like for human versus generated text.