Greedy Decoding

At every generation step a language model hands you a full probability distribution over its vocabulary, tens of thousands of numbers. Greedy decoding does the least imaginative thing possible with that distribution: it picks the single highest-probability token, appends it to the sequence, and repeats. No search, no randomness, no lookahead. It is the baseline every other decoding strategy is defined in contrast to, and it is still the right choice more often than its reputation suggests.

The mechanics

At step t, the model produces logits over the vocabulary, softmax turns them into probabilities p(x_t | x_1, ..., x_{t-1}) (see softmax-logits), and greedy decoding takes:

x_t = argmax_v  p(v | x_1, ..., x_{t-1})

That chosen token is fed back in as context for the next step, exactly as in any autoregressive-generation loop. There is no randomness in the choice itself, so with a fixed model, fixed weights, and fixed prompt, greedy decoding is close to deterministic (see decoding-determinism for the caveats that "close to" is hiding). It is also cheap: no candidate bookkeeping, no sampling overhead, just one argmax per step.

Why "locally optimal" is not "globally good"

The core problem is that greedy decoding never reconsiders. Suppose the model assigns token A probability 0.51 and token B probability 0.49 at step 3, but every continuation that follows B is far more probable, taken together, than every continuation that follows A. Greedy picks A anyway, because it only ever looks one step ahead. Once A is chosen it is locked into the context permanently; there is no mechanism to backtrack and try B. This is exactly the failure mode that motivated beam-search: keep more than one candidate alive so a locally weak choice doesn't foreclose a globally strong sequence.

In practice this shows up as a specific texture of failure: greedy decoding tends to loop. Once a phrase has been produced, the context that follows it makes repeating that phrase the highest-probability continuation again, and greedy has no penalty mechanism to break the cycle (see repetition-penalty). "The weather today is nice. The weather today is nice. The weather today is..." is the caricature, and it is a real, observed behaviour, not a strawman; it is one of the degenerate patterns catalogued in the broader literature on why likelihood-maximising decoders misbehave (see neural-text-degeneration).

Where it is the right call

Greedy decoding is not a lesser sampling method waiting to be upgraded, it is the correct choice whenever there is a single right answer and the model is well-calibrated toward it. Classification-style generation (emit a label, a yes/no, a short structured field), arithmetic and code where variance actively hurts, and tool-call argument generation are all cases where you want the model's single best guess, not a randomised draw from its distribution. Reasoning-tuned models are commonly run near-greedy for the same reason: an unlucky low-probability token early in a long chain of thought can derail everything downstream, and sampling noise compounds across steps in a way it does not in a single short answer.

Greedy is also the reference point for evaluating other decoders. If a paper reports that nucleus sampling produces more diverse text than greedy, the comparison is meaningful because greedy has a precise, reproducible definition: it is not "a temperature that happens to be low", it is the literal argmax.

When it falls down

• Myopic by construction. It cannot recover from an early suboptimal token because there is no mechanism to reconsider past choices; the entire search space it explores is one path wide.
• Degenerates into loops. Without an explicit repetition penalty or truncation-plus-sampling, greedy output on open-ended prompts frequently collapses into repeated phrases, because the highest-probability continuation of a repeated phrase is often the same phrase again.
• Zero diversity. Ten calls with the same prompt produce the same output (modulo hardware nondeterminism), which is fatal for any task, brainstorming, creative writing, data augmentation, that needs varied samples.
• High probability is not high quality. Greedy maximises per-step likelihood, and likelihood-maximising text is measurably duller and more repetitive than human text at the sequence level; picking the "most probable" token at every step is not the same as picking the best one.

Further reading

• Holtzman et al., 2019, The Curious Case of Neural Text Degeneration, arXiv:1904.09751 - documents how likelihood-maximising decoding (greedy and beam search) produces repetitive, low-quality text compared to sampling.
• Radford et al., 2019, Language Models are Unsupervised Multitask Learners (GPT-2) - early large-scale documentation of greedy/beam degeneration that motivated the shift toward sampling-based decoders.