Next-Token Prediction and Cross-Entropy Loss

Every capability a large language model has - translation, code, arithmetic, dialogue - is a side effect of one objective it was trained on: given a sequence of tokens, predict the next one. There is no separate "reasoning loss" or "translation loss" in pretraining. There is only next-token prediction, run over trillions of tokens, and everything else emerges because predicting the next token well enough requires modelling the structure that produced the text.

The objective, stated precisely

At each position, the model outputs a probability distribution over the whole vocabulary for the token that comes next. Training maximises the probability the model assigns to the token that actually came next. Equivalently, and more usefully for optimisation, it minimises the negative log of that probability, summed over the sequence:

loss = - (1/N) * sum_t  log P_model(x_t | x_1, ..., x_{t-1})

This is the cross-entropy between the model's predicted distribution and the one-hot "truth" (all mass on the real next token). Because the truth is one-hot, cross-entropy collapses to just the negative log-probability of the correct token. Assign the right token probability 0.9 and you pay -log(0.9) = 0.10; assign it 0.01 and you pay 4.6. The loss punishes confident wrongness far more than mild uncertainty, which is exactly the gradient signal you want.

Why log, and why it is the right shape

Two properties make log-loss the natural choice. First, it is the maximum-likelihood objective: minimising it is identical to maximising the probability of the training corpus under the model, so the model is literally trained to find the data most probable. Second, the gradient is clean. For a softmax output, the gradient of cross-entropy with respect to the logits is just predicted_probability - true_probability, a simple difference that flows backward without saturating (see sampling-decoding for what happens to those same logits at inference).

Teacher forcing and the shift

During training the model predicts every position in parallel, and at each position it is fed the true previous tokens, not its own predictions. This is teacher forcing, and it is why training is fast: one forward pass scores all N next-token predictions at once. The labels are just the input shifted left by one. A subtlety follows from this: the model never sees its own mistakes during training, which is part of why errors can compound at generation time when it must consume its own output (see autoregressive-generation).

When it falls down

• Loss is not capability. A lower cross-entropy means better next-token prediction on the data distribution, not better reasoning. Two checkpoints with near-identical loss can differ sharply on downstream tasks; the metric is a proxy.
• Tokenisation changes the loss. Cross-entropy is per-token, so a corpus that tokenises into more pieces (see tokenisation-bpe) is not directly comparable. This is why perplexity comparisons across different tokenisers are meaningless (see perplexity-language-models).
• Uniform weighting. Every token contributes equally, so the model spends as much capacity learning to predict the second half of a common word as it does a genuinely hard factual continuation. Most tokens are easy, which is why loss curves flatten quickly and the interesting learning hides in the long tail.

Further reading

• Bengio et al., 2003, A Neural Probabilistic Language Model - the neural next-token objective, before transformers.
• Radford et al., 2019, Language Models are Unsupervised Multitask Learners (GPT-2) - the argument that pure next-token prediction at scale yields general capability.