Cross-Entropy and KL Divergence

Every language model is trained on something called "cross-entropy loss," and most people who use that phrase daily could not say what is cross about it. There are two distributions involved: p, the true distribution the data came from, and q, the model's predicted distribution. Cross-entropy measures the cost of using q's code to describe samples that actually came from p. Understanding that gap precisely explains why pretraining, distillation, and RLHF all reach for the same handful of formulas.

The decomposition

H(p, q) = H(p) + KL(p || q)

Cross-entropy equals the true entropy of the data plus the KL divergence from the model to the data. H(p) is a fixed property of the data, the model cannot change it. So minimising cross-entropy over q is exactly, term for term, minimising KL(p || q). This is not an analogy; it is algebra. Training a language model is, precisely, driving the KL divergence between the model's predictions and the true (unknown, empirical) data distribution toward zero, and the entropy floor from entropy-and-surprise sets the best possible cross-entropy you could ever reach.

Excess bits, concretely

KL(p || q) has a clean reading: it is the number of extra bits you pay, per symbol on average, by using a code built for q when the true distribution is p. If p puts 90% of its mass on outcome A and q puts only 50% there, every time A actually occurs you are paying more bits than an optimal code would, because your code was built for the wrong distribution. KL(p||q) is zero only when q = p exactly, and it is always non-negative, a fact provable directly from Jensen's inequality.

Forward KL versus reverse KL

KL divergence is not symmetric, and the direction you minimise changes what kind of mistake the model prefers to make.

• Forward KL, KL(p_data || p_model), is what maximum-likelihood training minimises (see maximum-likelihood-estimation). It penalises the model heavily for assigning near-zero probability anywhere the data has real probability mass. The model is forced to spread some probability everywhere it has ever seen data, which produces broad, mode-covering, sometimes hedgy distributions, exactly the behaviour of a pretrained base model that "wants" to keep every plausible continuation alive.
• Reverse KL, KL(p_model || p_reference), is what RLHF's KL penalty term typically minimises when it keeps a fine-tuned policy anchored to a reference model. It penalises the model for putting probability where the reference has almost none, but is comparatively cheap about abandoning regions the reference covers that the model does not care about. This produces mode-seeking behaviour: the policy commits confidently to specific high-reward regions instead of hedging across everything the reference considered plausible. It is why RLHF'd models sound more decisive than their base model, and also why over-aggressive reverse-KL regularisation can collapse a policy onto a narrow set of stylistic patterns.

Distillation: both terms in one loss

Knowledge distillation trains a small "student" model to imitate a large "teacher." The standard loss blends a cross-entropy term against the true hard labels with a KL term against the teacher's softened output distribution (softened by dividing teacher logits by a temperature greater than 1 before softmax, spreading probability across near-miss classes so the student learns how wrong the wrong answers are, not just which one is right). The KL term is what actually transfers the teacher's "dark knowledge," the relative probabilities it assigns to tokens it did not pick, which a hard-label cross-entropy loss alone throws away entirely.

When it falls down

• KL explodes at zero. If q assigns literally zero probability to an event p treats as possible, KL(p||q) is infinite. This is exactly why label smoothing and additive smoothing exist: they keep no probability at a hard zero so the loss stays finite and gradients stay usable.
• A single scalar hides wild per-token variance. Aggregate cross-entropy is an average; the actual KL contribution per token can vary by orders of magnitude between an easy continuation and a genuinely hard one, and the scalar loss cannot tell you which.
• KL is not a distance. It fails the triangle inequality and is asymmetric, so statements like "model A is closer to the data than model B" require specifying which direction of KL you mean; swapping the arguments can reverse the conclusion.

Further reading

• Hinton, Vinyals, Dean, 2015, Distilling the Knowledge in a Neural Network, arXiv:1503.02531 - the paper that formalised soft-label distillation via KL.
• Rafailov et al., 2023, Direct Preference Optimization, arXiv:2305.18290 - shows the reverse-KL-regularised RLHF objective can be solved in closed form without an explicit reward model.