Nucleus (Top-p) Sampling

Top-k-sampling's central weakness is that a fixed candidate count cannot fit every distribution shape a model produces. Nucleus sampling, introduced by Holtzman et al., 2019 in "The Curious Case of Neural Text Degeneration", fixes this by truncating on cumulative probability mass instead of rank. It has been the long-standing default decoding strategy for open-ended generation across most major chat systems.

The mechanism

Sort tokens by probability, descending, and accumulate their probabilities until the running sum first exceeds a threshold p (commonly 0.9 to 0.95). That set of tokens, however many of them it takes, is the "nucleus":

nucleus = smallest set S such that sum_{i in S} p_i >= p

Zero out everything outside the nucleus, renormalise, and sample. The crucial difference from top-k is that the size of S is not fixed; it is whatever the distribution's shape requires to reach p of the probability mass.

Why cumulative mass fixes the sizing problem

Consider the same two cases that broke a fixed k. When the model is highly confident, one token might carry 0.95 of the probability mass on its own; with p=0.9 the nucleus might contain just that single token (or two or three), because cumulative probability already exceeds the threshold almost immediately. Top-p naturally shrinks to a tiny, sharp candidate set exactly when the model is sure of itself. When the model is genuinely uncertain and probability is spread thin across dozens of plausible tokens, none individually large, it takes many tokens to accumulate 0.9 of the mass, so the nucleus grows to include all of them. Top-p naturally widens exactly when the model's own distribution says width is warranted.

This is the paper's core argument, stated more precisely: the size of the reasonable candidate set is itself a property of the distribution at that step, not a constant, and any fixed-size truncation (top-k, or no truncation at all) is fighting that fact rather than using it.

Degeneration: the problem nucleus sampling was built to solve

Holtzman et al. frame top-p within a larger diagnosis. They show that maximisation-based decoding, greedy-decoding and beam-search, produces text that is measurably duller than human text: lower lexical diversity, a higher rate of repeated n-grams, and a probability distribution over generated text that concentrates far more sharply than the distribution of human-written text does. Pure ancestral sampling from the full, untruncated distribution (see temperature-sampling at T=1 with no truncation) has the opposite problem: it too readily samples from the long unreliable tail, producing incoherent output. Nucleus sampling sits between these failure modes, sampling genuinely from the model's distribution (unlike beam search, which maximises), but only from the part of that distribution the model actually believes in (unlike raw sampling, which does not truncate at all). See neural-text-degeneration for the fuller diagnosis this concept only summarises.

Practical tuning

p=0.9 to p=0.95 are the common defaults, and top-p is nearly always paired with a temperature setting rather than used alone, temperature shapes how sharply probability is distributed within the nucleus, top-p decides how large the nucleus is. Lower p (0.7 to 0.85) narrows toward more conservative, focused output; p near 1.0 approaches unfiltered sampling and reintroduces the long-tail incoherence problem top-p exists to solve.

When it falls down

• Still ignores the shape within the nucleus. Top-p defines which tokens are eligible but says nothing about how peaked or flat the distribution is among them; two nuclei of the same cumulative-probability size can have wildly different internal shapes, one dominated by a single token, another nearly uniform, and top-p treats them the same. This is part of what motivated min-p-and-typical-sampling.
• Can behave oddly at very high temperature. Raising temperature before applying top-p flattens the distribution, which can let the nucleus grow to include tokens that were originally far into the unreliable tail, an interaction min-p was specifically designed to avoid.
• p is task-sensitive, not universal. A p tuned for chat can be too loose for structured extraction and too tight for creative brainstorming; like temperature, it is a task setting, not a model constant.
• Doesn't prevent repetition on its own. Truncating the tail does not stop the model from confidently, repeatedly selecting the same short phrase if that phrase remains the highest-probability nucleus member at each recurrence; that failure mode needs an explicit fix (see repetition-penalty).

Further reading

• Holtzman et al., 2019, The Curious Case of Neural Text Degeneration, arXiv:1904.09751 - the original nucleus sampling paper and the fullest diagnosis of why maximisation-based decoding degenerates.
• Fan, Lewis, Dauphin, 2018, Hierarchical Neural Story Generation, arXiv:1805.04833 - the top-k baseline that nucleus sampling was built to improve on.