Entropy and Surprise

"The pope announced " is easy to finish. "Colourless green ideas " is not. Both are five-word prompts, but one leaves almost no uncertainty about the next token and the other leaves a lot. Information theory gives that difference an exact number instead of a vibe: the less predictable an event, the more bits it costs to describe, and entropy is what you get when you average that cost over every possible next token. This concept is about the intuition and the units; for the formal probability machinery behind it, see probability-information-theory.

Self-information: turning probability into bits

Define the surprise of an outcome x with probability p(x) as:

self_information(x) = -log2 p(x)

Every time you halve the probability of an event, you add exactly one bit of surprise. An event with probability 1 (certain) has zero surprise; an event with probability 1/1000 costs about 10 bits. This is not a metaphor borrowed from computing, it is the literal number of yes/no questions an optimal guesser needs, on average, to pin the event down. A next-token distribution where the true token gets probability 0.5 costs 1 bit; a distribution that assigns it 0.001 costs about 10 bits, ten times the code length for a token the model considered ten times less likely, in log space.

Entropy is just the expectation of self-information over the whole distribution:

H(X) = sum_x  p(x) * -log2 p(x)

Average bits of surprise per symbol, weighted by how often each symbol actually occurs.

The Shannon guessing game

In 1951, Claude Shannon wanted to know how much real entropy printed English actually has, so he ran an experiment: show a person the preceding text and have them guess the next character, one guess at a time, recording how many guesses it took. A perfectly random 27-symbol alphabet (26 letters plus space) has a maximum entropy of about 4.75 bits per character. Shannon's human guessers needed far fewer: his estimate came out around 1 to 1.5 bits per character. English is enormously redundant, most of what looks like information in a sentence is actually predictable from what came before, and only a small fraction of each character carries genuinely new content. This experiment is the ancestor of every perplexity number reported today; a modern LLM's cross-entropy on English text is a far more precise, far more automatable version of the same guessing game (see perplexity-language-models).

Bits versus nats

Deep learning frameworks almost universally use natural log (ln), not log2, when computing loss, because the derivative of ln is simpler to work with and matches the exponential family math used throughout probability. The resulting unit is the nat rather than the bit. Converting is one multiplication:

bits = nats / ln(2)     # ln(2) ~= 0.693, so 1 nat ~= 1.443 bits

When a training log reports "loss: 2.1", that is 2.1 nats per token, equivalent to about 3.0 bits per token. Keep the unit straight when comparing numbers from different sources; a paper reporting entropy in bits and one reporting loss in nats can look like they disagree by 44% when they agree exactly.

Entropy is a floor, not a target

The source coding theorem says the entropy H(X) of a distribution is the minimum expected number of bits any lossless code can use to represent samples from it, and that minimum is achievable (up to rounding) by an optimal code matched exactly to the true distribution. No language model, however good, can drive its cross-entropy loss below the true entropy of the text it is modelling, because that entropy is a property of the data-generating process, not of the model. Training only closes the gap between the model's cross-entropy and that floor; it never gets to move the floor itself. This is the idea that cross-entropy-and-kl and language-modelling-as-compression both build directly on top of.

When it falls down

• You never actually know the true entropy. Every "entropy of English" number, Shannon's guessing game, n-gram counting, or a modern LLM's cross-entropy, is an estimate under some methodology, and different methodologies disagree by a factor of two or more. There is no ground truth to check against.
• Entropy is an average, not a guarantee. A distribution can have low average entropy while containing rare, wildly unpredictable tokens (names, numbers, dates), and those rare high-surprise tokens are disproportionately where language models get things wrong.
• Entropy depends on how much context you condition on. Conditional entropy shrinks as you add more context, so a bare claim like "the entropy of a token is 1.3 bits" is meaningless without stating how much preceding text the estimate assumes.

Further reading

• Shannon, 1948, A Mathematical Theory of Communication - the paper that defines entropy and the source coding theorem.
• Shannon, 1951, Prediction and Entropy of Printed English - the human guessing-game estimate of English's entropy.