Logit Bias and Masking

Every decoding technique covered elsewhere in this bundle, temperature, top-k, top-p, repetition penalties, operates on the shape of the distribution generically, without knowing anything about what the tokens actually mean. Logit bias and masking are the opposite: a direct, targeted edit to specific tokens' logits, applied before the truncation and sampling machinery runs. They are the mechanism behind two very different guarantees, "make this token more or less likely" and "this token cannot appear," and conflating the two is a common source of subtle bugs.

Additive logit bias: nudging, not forcing

A logit bias adds a constant to specific tokens' logits before softmax:

logit_i' = logit_i + bias_i

Because softmax is exponential, even a moderate additive bias has an outsized multiplicative effect on the resulting probability (recall from softmax-logits that a logit gap of 2 corresponds to roughly a 7.4x probability ratio). A bias of +2 on a token makes it substantially more likely to be sampled; a bias of -100 makes it, in practice, unsamplable without formally forbidding it, since exp(-100) underflows to zero in floating point long before the true -infinity would.

The key property of an additive bias, contrasted with a hard mask, is that it is a nudge, not a guarantee. A token biased down heavily can still theoretically be sampled if every other token's probability is even more suppressed by other factors; a token biased up is more likely, not certain, to appear. This makes logit bias the right tool for steering, discouraging a particular word choice, encouraging a persona's vocabulary, without absolutely forbidding anything, which matters when the model might legitimately need that token in some other context.

Hard masking: the guarantee

Masking sets specific logits to negative infinity (or, in floating point, the most negative representable value, which after softmax rounds to exactly zero probability) before any other decoding step runs:

logit_i = -infinity   if i in forbidden_set

Unlike a bias, this is absolute: the token cannot be sampled, full stop, regardless of temperature, top-k, or top-p settings applied afterward, because zero probability stays zero probability through any renormalisation. This is the mechanism underneath structured-generation-constrained-decoding: to guarantee valid JSON, a valid enum value, or output matching a grammar, the decoder computes, at every step, the set of tokens that would violate the target structure, and masks all of them to -infinity before sampling proceeds. The guarantee is only as strong as the mask is complete; a masking scheme that misses an edge case (an escaped character sequence in a JSON string, say) reintroduces exactly the failure it was meant to prevent.

Order of operations matters

Whether bias and masking are applied before or after temperature and truncation changes behaviour materially. Masking should logically happen first, since a token that must be excluded should never re-enter consideration regardless of how temperature reshapes what remains, and reputable constrained-decoding implementations enforce this ordering strictly. Additive bias is more ambiguous: applying it before temperature means the bias itself gets rescaled by T (a large bias at high temperature has a proportionally smaller effect after division), while applying it after temperature keeps the bias's absolute effect constant regardless of sampling settings. Neither convention is universally "correct"; different inference APIs make different choices, and the same nominal logit_bias value can produce visibly different behaviour across providers as a result.

Common uses

Logit bias is the mechanism behind a range of practical features that have nothing to do with quality or diversity per se: banning a specific profanity list, discouraging a model from mentioning a competitor's name, nudging toward a required disclaimer phrase, or (at the extreme, full masking) implementing a strict allow-list decoder that can only ever emit tokens from a fixed small vocabulary, useful for classification-style generation where the answer must be one of a known set of labels.

When it falls down

• Tokenisation misalignment. A word you want to bias or ban may not correspond to a single token; it might be several tokens, or the same word might tokenise differently depending on leading whitespace or capitalisation (see tokenisation-bpe). Biasing "the token for X" often requires biasing several token IDs to have the intended effect at all.
• A bias is not a ban. Treating a large negative additive bias as equivalent to a hard mask is a common mistake; under unusual distributions (every other token suppressed further by some other factor) a heavily-biased-down token can still, in principle, be sampled.
• Masking can be incomplete. Constrained decoding is only as reliable as the grammar-to-mask translation; subtle grammar edge cases (nested quotes, escape sequences, multi-token literals) can leak invalid tokens through an incomplete mask.
• Order-of-operations is not standardised across providers. Whether bias applies before or after temperature/truncation differs by implementation, so the same nominal bias value is not portable across inference stacks without checking documentation.

Further reading

• OpenAI API documentation, logit_bias - a widely used production implementation of additive logit bias, with its documented value range and effect.
• Willard and Louf, 2023, Efficient Guided Generation for Large Language Models, arXiv:2307.09702 - a formal treatment of mask construction for grammar-constrained decoding, the hard-masking side of this concept at scale.