Determinism and Reproducibility in Decoding

"Set the seed and set temperature to zero" is the standard advice for reproducible LLM output, and it is necessary but not sufficient. Teams that rely on bit-identical output for testing, evaluation, or caching regularly discover that the same prompt, same model, same seed, and same temperature-zero setting still produces different completions across runs. The gap between the intuition ("deterministic decoding should be deterministic") and the reality is worth understanding precisely, because it changes what you can and cannot rely on.

What determinism actually requires

A decoding process is only as deterministic as every step feeding into it. Greedy decoding (greedy-decoding) removes randomness from the token selection step, argmax has no random component. Setting a random seed removes randomness from any sampling step (temperature, top-k, top-p all rely on a random number generator once truncation is applied; see temperature-sampling). Together, these should, in principle, produce identical output on identical input. In practice, a third source of nondeterminism sits underneath both: the forward pass that produces the logits in the first place is not always bit-identical across runs, even with identical weights and identical input.

Where the real nondeterminism lives

Floating-point addition is not associative: (a + b) + c can differ in its last bits from a + (b + c) due to rounding at each step. GPU kernels for matrix multiplication and reduction operations (the operations that dominate a transformer forward pass) do not guarantee a fixed order of summation, particularly under batched inference, where the specific batch composition, which other requests happen to be running alongside yours, can change how work is partitioned across threads and therefore the order floating-point reductions happen in. Two calls with identical prompt, identical seed, identical temperature-zero setting can still diverge if they land in different batches, on different hardware, or under a different kernel scheduling decision, because the logits themselves differ in their last few bits before decoding ever sees them. Most of the time this changes nothing, argmax of 0.4821 and argmax of 0.48213 picks the same token, but near a genuine tie between two tokens, a last-bit difference in the logits is enough to flip which one wins, and that single flipped token changes every token generated after it, since autoregressive-generation feeds each choice back in as context.

This is a distinct phenomenon from decoding-strategy randomness, and it cannot be fixed by seeding the sampler, because the nondeterminism is upstream of sampling, in the numerics of the forward pass itself (see numerical-computation-gotchas if covered elsewhere in the curriculum for the general version of this problem).

What actually helps

Running on identical hardware with an identical batch size, and using kernels explicitly built for deterministic reduction order (some inference frameworks expose a "deterministic mode" flag that forces a fixed reduction order at a real throughput cost), narrows the gap substantially. Fixing the random seed guarantees the sampling step is reproducible given identical logits. Neither addresses the batching-dependent kernel nondeterminism directly, and most production serving stacks, which batch requests dynamically for throughput, do not offer a practical way to eliminate it without sacrificing the batching that makes serving affordable in the first place.

Why this matters beyond curiosity

Evaluation pipelines that assume "temperature 0 means I can diff two runs" will intermittently fail in ways that look like flaky tests but are not; the flakiness is a real property of the serving stack, not a bug in the eval. Caching layers that key on (prompt, temperature=0) and expect a cache hit to always be valid for reuse are making the same unwarranted assumption. Regression testing for prompt changes needs either a tolerance for near-tie divergence or a controlled, unbatched, fixed-hardware evaluation harness, an accommodation many teams learn about only after chasing a nondeterminism bug that "shouldn't be possible."

When it falls down

• "Temperature 0" is not a determinism guarantee, it is a variance reduction. It removes sampling randomness but not forward-pass numerical nondeterminism from batched, parallel floating-point reduction.
• Seeding only covers the sampler. A fixed seed makes the random draw reproducible given fixed logits; it says nothing about whether the logits themselves are reproducible across runs, hardware, or batch compositions.
• Deterministic kernels cost throughput. Forcing a fixed reduction order to eliminate batching-dependent drift generally means giving up the batching flexibility that keeps serving costs down, a real tradeoff, not a free correctness fix.
• The effect is rare per token but compounds over a sequence. A single flipped near-tie token is a small probability event, but autoregressive generation means one flip early in a long sequence propagates into every subsequent token, so the practical divergence rate over long generations is higher than the per-token flip rate alone would suggest.

Further reading

• PyTorch, Reproducibility - official documentation of the floating-point, CUDA-kernel, and batching sources of nondeterminism that seeding alone cannot fix; the general pattern applies beyond PyTorch to any batched GPU inference stack.
• Holtzman et al., 2019, The Curious Case of Neural Text Degeneration, arXiv:1904.09751 - the decoding-strategy side of variance (sampling versus maximisation), distinct from but often confused with the numerical nondeterminism covered here.