Repetition Penalty and No-Repeat N-Grams

Truncation and temperature control which tokens are eligible and how sharply the distribution is sampled, but neither one has any memory of what has already been generated. A decoder can truncate perfectly and still loop, because a repeated phrase can remain the highest-probability (or nucleus-eligible) continuation every time the context recurs. Repetition control is the family of techniques that adds that missing memory.

Repetition penalty: discount tokens by prior appearance

The most common approach, popularised in the Hugging Face transformers generation utilities and CTRL-style conditional generation, subtracts a fixed penalty from the logits of every token that has already appeared in the generated sequence, before sampling:

if token already generated:
    logit = logit / penalty   if logit > 0
    logit = logit * penalty   if logit <= 0

A penalty greater than 1.0 makes previously used tokens less likely to be picked again; penalty = 1.0 is a no-op. The division-versus-multiplication split (rather than a flat subtraction) exists to handle positive and negative logits consistently: dividing a positive logit shrinks it, dividing a negative logit would make it larger (less negative), so the sign is flipped to multiplication for negative logits to keep the penalty's direction consistent regardless of a token's raw logit sign.

A related pair of parameters, common in commercial chat APIs, separates the penalty into frequency penalty (scales with how many times a token has already appeared, so the third repeat is discouraged more than the first) and presence penalty (a flat penalty applied the moment a token has appeared at all, regardless of count). Frequency penalty targets loops of the same token or phrase recurring many times; presence penalty targets topic fixation, nudging the model to introduce new vocabulary rather than circling the same handful of words.

No-repeat n-grams: a hard ban

A blunter and more absolute tool: forbid the model from generating any n-gram (commonly n=3 or n=4) that has already appeared verbatim earlier in the sequence. Implemented by masking to negative infinity any token whose selection would recreate a previously seen n-gram:

if (last n-1 tokens + candidate token) in already_seen_ngrams:
    logit = -infinity

Unlike a soft penalty, this is not a matter of degree, the banned continuation becomes literally impossible, regardless of how confident the model is. This is standard in beam search implementations especially, where it directly counters beam search's tendency toward exact repeated phrases (see beam-search).

The dial you can turn too far

Both mechanisms share the same failure shape: they cannot distinguish problematic repetition from repetition that is semantically necessary. A soft repetition penalty set too high starts discouraging function words (articles, common verbs) that any coherent English sentence must reuse constantly, and pushes generation toward strained, evasive phrasing that avoids saying the same noun twice even when clarity demands it. No-repeat n-gram bans are worse in this respect precisely because they are absolute: a legitimate reason to repeat a 3-gram, a person's name mentioned twice, a technical term with no synonym, a required boilerplate phrase, is not accommodated; the model is simply forbidden from using it again for the rest of the generation, which is a real and reported failure mode in long technical or legal text generation.

The fix in practice is usually a smaller penalty and a shorter, more targeted n-gram window, not a larger one; over-correcting for repetition trades one visible failure (looping) for a subtler one (evasive, unnatural phrasing) that is harder to notice in a quick read.

When it falls down

• Penalises necessary repetition indiscriminately. Named entities, technical terms, and required boilerplate get suppressed along with genuine loops; neither mechanism has any notion of semantic necessity.
• No-repeat n-grams are absolute, not tunable in degree. Once an n-gram is banned it is banned for the rest of the sequence regardless of how strongly the model wants to use it, which is a much blunter instrument than a soft logit penalty.
• Interacts with truncation order. Whether the penalty is applied before or after top-k/top-p truncation changes the outcome, since a penalised token might already have been excluded by truncation, or might newly fall below the truncation threshold as a result of the penalty; implementations differ.
• Doesn't fix the underlying cause. Repetition penalties treat the symptom (the model wants to loop) rather than the cause (the context has become locally self-reinforcing); they mask degenerate behaviour rather than resolving why a well-truncated, well-tempered decoder still drifts toward it on some prompts.

Further reading

• Keskar et al., 2019, CTRL: A Conditional Transformer Language Model for Controllable Generation, arXiv:1909.05858 - popularised the multiplicative repetition penalty used in most current generation libraries.
• Holtzman et al., 2019, The Curious Case of Neural Text Degeneration, arXiv:1904.09751 - documents repetition as a defining symptom of likelihood-maximising decoding, the broader problem repetition penalties patch over.