Bits-per-Byte

The perplexity-language-models concept ends with a warning: perplexity is measured per token, and tokens are not a standard unit, a model with a coarser tokeniser can post a lower perplexity while modelling language no better at all. Bits-per-byte (BPB) is the fix. It reports how many bits, on average, the model needs to encode each raw UTF-8 byte of text, and every model, regardless of vocabulary or tokeniser, is measured against the exact same underlying byte stream.

The formula

bits_per_byte = (total_loss_in_nats / ln(2)) / total_bytes_of_text

Equivalently, if you already have cross-entropy per token, multiply by the average number of tokens per byte for that tokeniser and text:

bpb = cross_entropy_bits_per_token * (num_tokens / num_bytes)

This is exactly the source-coding-theorem quantity from entropy-and-surprise, just expressed against a fixed physical unit (bytes) instead of a model-specific one (tokens).

Why bytes, not characters

"Character" sounds like the natural unit, but it is not universal. A Unicode code point is not a fixed number of bytes: an ASCII letter is 1 byte in UTF-8, a Hindi or Cyrillic character is typically 2 to 3 bytes, and many CJK (Chinese/Japanese/Korean) characters are 3 bytes. A model evaluated in bits-per-character would look artificially efficient on scripts where one "character" packs more raw information, and artificially inefficient on scripts where it does not. UTF-8 byte count sidesteps this: it is a fixed, language-agnostic, tokeniser-agnostic encoding that every piece of digital text already has, making it the only unit that lets you compare across languages and across models fairly.

A worked comparison

Suppose model A uses a 32k-token BPE vocabulary and model B uses a 128k-token vocabulary, both trained on the same corpus. Model B's larger vocabulary packs more raw text into each token, so B needs fewer tokens to cover the same sentence, and each of those tokens is intrinsically "worth more" information. B's cross-entropy per token will tend to look higher than A's even if B is a genuinely better language model, purely because each of B's tokens is doing more work. Converting both to bits-per-byte removes the illusion: it normalises away the tokeniser entirely and asks the only fair question, how many bits does each model need to reconstruct this exact sequence of bytes.

Where you will see it

Bits-per-byte shows up wherever cross-tokeniser or cross-corpus comparison matters: dataset papers reporting model quality across mixed-domain corpora, and evaluation harnesses that need a single number comparable across model families with different vocabularies. It is also the natural metric for treating language modelling explicitly as compression, the framing explored in language-modelling-as-compression.

When it falls down

• It still depends on the evaluation corpus. BPB normalises away the tokeniser, not the text. A model tuned on English news will show a worse bpb on source code or a non-Latin script, so a single bpb number without a stated corpus is exactly as uninterpretable as an unstated-corpus perplexity number.
• It is easy to compute wrong. The conversion needs the exact untokenised byte length of the evaluation text. A common implementation bug computes bpb using token counts as a proxy for byte counts, which silently reintroduces the tokeniser dependence bpb exists to remove.
• It rewards compressibility, not usefulness, the same caveat perplexity carries. A model can achieve an excellent bpb on repetitive boilerplate text and a mediocre bpb on short, information-dense text, without that difference telling you anything about task performance.

Further reading

• Gao et al., 2020, The Pile: An 800GB Dataset of Diverse Text for Language Modeling, arXiv:2101.00027 - reports bits-per-byte across a deliberately heterogeneous, multi-domain corpus.
• Kaplan et al., 2020, Scaling Laws for Neural Language Models, arXiv:2001.08361 - the scaling-law paper that established loss-vs-compute measurement conventions this metric builds on.