Temperature Sampling

Once you decide to sample from a model's output distribution rather than always taking the argmax, you need a way to control how "risky" the sampling is. Temperature is the simplest such control: a single scalar, applied before softmax, that reshapes the distribution's sharpness without changing which token is most likely.

The mechanism

Given logits z, temperature T divides every logit before the softmax normalisation:

p_i = exp(z_i / T) / sum_j exp(z_j / T)

Because every logit is divided by the same T, the ranking of tokens by probability never changes, only the spread. T = 1 recovers the model's raw distribution unchanged. T < 1 divides by a number less than one, which means dividing by a fraction multiplies the effective logit gaps, pulling probability mass toward the already-likely tokens (see softmax-logits for why the exponential turns additive logit gaps into multiplicative probability ratios, which is exactly what temperature is exploiting). T > 1 compresses the gaps, flattening the distribution toward uniform. As T -> 0, the distribution collapses onto the single most probable token, so temperature-0 sampling and greedy-decoding converge (with the caveat that many implementations special-case T=0 to literal argmax, since dividing by zero is undefined).

Why it is a global rescale, not a targeted fix

Temperature's defining limitation is that it treats the whole distribution uniformly. It has no notion of "good" tokens versus "bad" tokens beyond what the model's own probabilities already encode; it only stretches or compresses the gaps that are already there. Raise temperature to get more diverse, less predictable output, and you get exactly that, but you also proportionally raise the probability of every token in the long tail, including the ones that are outright wrong or incoherent. A model's probability distribution over a 50,000-token vocabulary typically has a short list of genuinely reasonable next tokens and an enormous tail of implausible ones with tiny but nonzero probability; softmax never assigns exact zero (see softmax-logits). High temperature does not distinguish "creative but sensible alternative" from "nonsense token that happened to get 0.0001 probability" and amplifies both together.

This is precisely why temperature is almost never used alone in production systems. It is paired with a truncation step, top-k-sampling, nucleus-sampling-top-p, or min-p-and-typical-sampling, that first removes the incoherent tail, and only then does temperature reshape what remains. Temperature answers "how much should I flatten the distribution I'm sampling from"; truncation answers "which part of the distribution is even worth considering." Conflating the two is a common source of "I raised the temperature and now it's just gibberish" bug reports.

Choosing a value in practice

There is no universal correct temperature; it is a property of the task, not the model. Factual question-answering and code generation typically want low temperature (0.0 to 0.3), because variance introduces errors without adding value the task can use. Open-ended creative writing or brainstorming benefits from moderate to higher temperature (0.7 to 1.2), where the point is genuinely to explore the distribution's breadth. Values much above 1.5 to 2 are rarely useful on an untouched distribution; without truncation, the tail overwhelms the signal well before then.

When it falls down

• Not a safety knob. Lowering temperature reduces variance, it does not guarantee correctness or filter unsafe content; a low-temperature model can still confidently emit a wrong or harmful answer if that answer is simply the model's highest-probability output.
• Interacts badly with a peaked distribution. If the model is already extremely confident (one token near probability 1), moderate temperature changes have almost no visible effect, because there is little gap left to stretch.
• Amplifies the tail as much as the head. Without truncation, raising temperature to get diversity proportionally raises nonsense along with genuine alternatives; this is the standard justification for combining it with top-k, top-p, or min-p.
• T=0 is a convention, not a law. Some serving stacks implement T=0 as true argmax; others implement it as a very small but nonzero temperature, which can still sample rare ties differently. Do not assume bit-identical output across providers at "temperature 0" (see decoding-determinism).

Further reading

• Ackley, Hinton, Sejnowski, 1985, A Learning Algorithm for Boltzmann Machines - the statistical-mechanics origin of "temperature" as a distribution-sharpening parameter, predating its use in language model decoding by decades.
• Holtzman et al., 2019, The Curious Case of Neural Text Degeneration, arXiv:1904.09751 - situates temperature within the broader truncate-then-sample decoding picture.