NLP Foundations
Autoregressive Generation
How a language model turns next-token prediction into a paragraph, and why generation is a loop that feeds its own output back as input.
beginner · 6 min read
A modern language model does exactly one thing: given a sequence of tokens, it predicts a probability distribution over the next token. Everything else, chat, code, essays, agent trajectories, is that single operation run in a loop. Understanding the loop is the difference between treating a model as a black box and reasoning about why it behaves the way it does.
The generation loop
Generation is autoregressive: each new token is appended to the input, and the model runs again on the longer sequence.
prompt -> model -> token_1
prompt + token_1 -> model -> token_2
prompt + token_1 + token_2 -> model -> token_3
...until a stop token or length limit
The model's own output becomes part of its next input. This is why a single early mistake can compound: token 3 is conditioned on token 2, so a wrong token does not just sit there, it reshapes everything that follows. The decoding strategy decides which token to actually pick from each distribution, and tokenisation decides what a "token" even is.
Teacher forcing vs generation
The mismatch that surprises people: during training the model always sees the correct previous tokens (teacher forcing), but during generation it sees its own previous tokens, which may be wrong. The model is never trained on its own mistakes, so at inference time it can wander into states its training never covered. This "exposure bias" is one reason long generations drift.
Why it costs what it costs
Autoregressive generation is inherently sequential: token N cannot be computed until token N-1 exists. You cannot parallelize across the output the way you can across the input. This is the root reason generation is latency-bound and why techniques like speculative decoding exist, to guess several tokens ahead and verify them in parallel. It is also why the KV cache matters: without caching, generating token 1000 would re-process all 999 prior tokens from scratch.
Where it shapes everything downstream
An agent is autoregressive generation with tools wired into the loop. The model emits tokens, some of those tokens are a tool call, the tool result is appended to the sequence, and the loop continues. Because every step conditions on the full accumulated sequence, the growth and quality of that sequence, its context, is the central engineering concern for anything long-running.