Maximum-Likelihood Estimation

Nowhere in a transformer's training loop does the math literally say "minimise cross-entropy." What it says is: find parameters that make the training corpus as probable as possible under the model. That is the definition of maximum-likelihood estimation (MLE), a principle far older than neural networks. Cross-entropy loss is simply what MLE looks like once you take a log and divide by corpus size; understanding the estimation theory underneath explains a handful of otherwise-mysterious facts, including where weight decay actually comes from.

The mechanics

Under teacher forcing, the model factorises sequence probability via the chain rule, and the likelihood of the whole training corpus is the product of every per-token conditional probability the model assigns to the token that actually appeared:

likelihood(theta) = product_t  P_theta(x_t | x_1, ..., x_{t-1})

Multiplying together thousands of numbers each less than 1 underflows to exactly 0 in floating point almost immediately, so in practice you always work in log space, where a product becomes a sum:

log_likelihood(theta) = sum_t  log P_theta(x_t | x_1, ..., x_{t-1})

Maximising this sum over theta is MLE. Negate it and divide by N, and it is exactly the cross-entropy loss from next-token-prediction-cross-entropy. "Minimise the loss" and "find the maximum-likelihood parameters" are the same optimisation problem written with opposite signs.

MLE is minimising forward KL to the data

As the training corpus grows, the empirical distribution of tokens observed in it approaches the true, unknown distribution that generated the language. Minimising cross-entropy against that empirical distribution is, asymptotically, minimising forward KL(p_data || p_model) (see cross-entropy-and-kl). This is the precise technical statement behind the intuition "more data gives a better fit": more data tightens the empirical distribution's approximation to the true one, which tightens the target the MLE optimisation is actually chasing.

MLE versus MAP: where weight decay comes from

Maximum a posteriori (MAP) estimation adds a prior belief over the parameters themselves:

maximize   log P(data | theta) + log P(theta)

Choose a zero-mean Gaussian prior over the weights, and log P(theta) becomes, up to a constant, -lambda * ||theta||^2, an L2 penalty on the weights. Weight decay is not an ad hoc regularisation trick bolted onto MLE; it is exactly MAP estimation with a Gaussian prior on the parameters. Any L2-regularised training run is, whether or not anyone frames it this way, doing Bayesian point estimation with a specific, if implicit, belief about how large the weights should be.

Consistency and what "more data helps" actually guarantees

Under classical statistical assumptions, MLE is consistent (given enough data drawn from the true distribution, the estimate converges to the true parameters) and asymptotically efficient (it achieves the lowest possible variance among unbiased estimators as data grows, the Cramer-Rao bound). Both guarantees assume the model family can actually represent the true data-generating distribution. Language was not generated by a transformer, so this assumption is false by construction. In practice, MLE on a misspecified model class still converges to something, the closest approximation to the truth reachable within that model family, measured in KL, but the clean statistical guarantees that make MLE attractive in a textbook do not strictly apply.

When it falls down

• No penalty for overconfidence. Given finite data, MLE happily assigns exactly zero probability to any valid continuation it never observed, the same sparse-data failure mode n-gram-models-and-smoothing exists to patch, and which neural models only avoid implicitly, through generalisation and techniques like label smoothing.
• Optimises the training distribution, not the deployment goal. A model that maximises likelihood on web text is optimising for "predict this corpus well," not for helpfulness, safety, or instruction-following. Closing that gap is exactly the job of RLHF and instruction tuning, which optimise a different objective entirely.
• Model misspecification breaks the nice guarantees. Because the true data process is never literally in the model family for language, MLE's consistency and efficiency results are aspirational, not literal, and the limit point MLE converges to is only "best available," not "correct" in any absolute sense.

Further reading

• Goodfellow, Bengio, Courville, 2016, Deep Learning, Ch. 5 (Maximum Likelihood Estimation) - the formal treatment, including the MLE-to-MAP connection.
• Murphy, Probabilistic Machine Learning: An Introduction - a free, modern treatment of MLE, MAP, and Bayesian estimation side by side.