Calibration of Language Models

A model is calibrated if, among every prediction it makes with 70% confidence, roughly 70% turn out to be correct. This is a different property from accuracy. A model can be highly accurate but badly calibrated (right answers stated with the wrong confidence), or calibrated but inaccurate (correctly saying "I'm only 30% sure" about something it does in fact get wrong 70% of the time). For anything that treats a model's probability output as a trust signal, reranking, abstention, hallucination flags, calibration is the property that actually matters, and it is not the same thing benchmark accuracy measures.

Measuring it: reliability diagrams and ECE

Bin a model's predictions by their stated confidence, then plot observed accuracy against stated confidence for each bin. A perfectly calibrated model traces the diagonal: predictions made at 80% confidence are correct 80% of the time, no more, no less. This plot is a reliability diagram. Summarised into one number, the weighted average gap between confidence and accuracy across bins is the Expected Calibration Error (ECE), lower is better calibrated. Guo et al., 2017 (arXiv:1706.04599) used exactly this measurement to show that modern, higher-capacity deep networks (using batch normalisation and greater depth than the networks calibration research had studied before) are systematically overconfident compared to older, smaller networks, even when their accuracy is higher. Their fix, temperature scaling, divides logits by a single learned scalar before softmax, sharpening or flattening the whole distribution without touching accuracy at all, and it closes most of the gap cheaply.

Pretraining is naturally calibrated; RLHF quietly breaks it

Kadavath et al., 2022 (arXiv:2207.05221) found that raw next-token probabilities from a pretrained LLM, evaluated on multiple-choice-style questions, are fairly well calibrated: the model's stated probability for an answer tracks its actual correctness rate reasonably closely, and this calibration tends to improve with model scale. Instruction tuning and RLHF change the picture. They optimise for what humans reward, not for likelihood on a data distribution, and confident-sounding answers tend to be rewarded regardless of whether the underlying probability estimate justifies that confidence. The result is a model whose raw internal logprobs may still be reasonably calibrated, but whose surface behaviour, both sampled outputs and any verbalised confidence statement, drifts toward overconfidence. This is why a base model's token probabilities are often a more honest confidence signal than a chat-tuned model saying "I'm 95% sure" in plain English.

Two different things both called "calibration"

It is worth separating two distinct properties that get the same name. Internal probability calibration asks whether the model's own token-level probability (its logprob for the answer it produced) matches empirical accuracy; it is measurable directly from logprobs but is not user-facing. Verbalised confidence calibration asks whether a natural-language confidence statement the model writes ("I'm fairly confident that...") matches empirical accuracy; this is a separately learned behaviour, generally worse calibrated than internal probabilities, and is the only kind of calibration a typical chat user ever sees. Anything downstream that treats an LLM's stated confidence as a reliability signal, selective prediction, abstention thresholds, hallucination detectors, needs to be explicit about which of the two it is actually measuring.