Language Modelling as Compression

Here is a claim stated precisely rather than loosely: a probabilistic model p(x) and a lossless compressor are the same mathematical object. You can convert either one into the other with zero loss of performance. This is not a metaphor borrowed to make information theory feel relevant to deep learning; it is a direct consequence of Shannon's source coding theorem, and it is the reason a large language model trained purely on text was able to out-compress purpose-built image and audio codecs it had never seen data from.

Arithmetic coding, in one paragraph

Arithmetic coding is the mechanism that makes the equivalence constructive rather than theoretical. Start with the interval [0, 1). For each symbol in the sequence, narrow the current interval into sub-intervals sized in proportion to the model's predicted probability for each possible next symbol, then keep only the sub-interval matching the symbol that actually occurred. A symbol the model thought was likely gets a wide sub-interval, costing few bits to specify; a symbol the model thought unlikely gets a narrow one, costing many. After the whole sequence, transmit any number inside the final tiny interval, and that number, decoded by running the identical model forward step by step, exactly reconstructs the original sequence. The number of bits this takes, on average, converges to the model's cross-entropy on the data (see entropy-and-surprise). A better predictive model is, by this construction, a better compressor, with no additional machinery required.

The Deletang et al. experiment

Deletang et al., 2023 (arXiv:2309.10668) made this concrete in a way that got attention well outside the compression-research community. They took language models trained purely on text and plugged them, unmodified, into an arithmetic coder as the probability engine, then fed the coder raw bytes of data the model had never trained on: ImageNet image patches and LibriSpeech audio clips. The result: a general-purpose LLM, evaluated purely as the predictive engine inside an arithmetic coder, compressed these out-of-domain images and audio more tightly than PNG and FLAC, codecs built specifically for those data types. The mechanism is exactly the one above: compression ratio is a direct function of predictive accuracy, and a large enough general-purpose predictor, trained on enough varied structure, can out-predict a small hand-engineered domain-specific model, even completely off-distribution.

Why this matters beyond a benchmark

The result is often cited as evidence for a stronger claim: that prediction, compression, and something like "understanding" might be three names for the same underlying capacity. This is not a new idea, it motivates the Hutter Prize, a standing reward for compressing a fixed sample of Wikipedia text, on the explicit premise that better compression requires better modelling of the structure and semantics of the text, not just better bit-shuffling. It also connects directly to the minimum description length (MDL) principle: given a choice between models that fit data equally well, the one with the shortest total description (model plus compressed residual) is the one that has captured genuine structure rather than memorised noise, an information-theoretic restatement of Occam's razor that shows up throughout model selection theory.