Token Fertility and Multilingual Fairness

Send the same paragraph of news text through a general-purpose tokeniser once in English and once in a lower-resource language, and the second version can come out at several times the token count for what is, by any human reading, the same amount of information. Researchers call this gap token fertility, and it is not a minor rounding error, it is a structural cost baked into the vocabulary before any user ever sends a request.

Defining fertility

Fertility, in this context, is the average number of tokens a tokeniser produces per reference unit of text, commonly measured per word or per UTF-8 byte, for a given language. A tokeniser trained on a corpus dominated by English and a handful of high-resource languages learns long, efficient merges for those languages' common words and morphology, while scripts and languages that were rare or absent in the training corpus never accumulate enough merges to compress efficiently, so their text stays closer to the byte-level base alphabet the whole way through (see byte-level-bpe). The resulting fertility ratio can differ by several times between the best-served and worst-served languages under the exact same fixed vocabulary.

Why this compounds into unfairness

Because commercial pricing and context windows are denominated in tokens, not words or bytes, high-fertility languages pay a real, measurable premium: more tokens for the same content means higher API cost, a smaller effective context window for the same document, and slower generation, since decode latency scales with output token count (see the-tokenisation-tax, context-windows-long-context, and autoregressive-generation). The disadvantage does not stop at inference. During pretraining, a fixed compute or token budget spent on high-fertility text covers less actual content per token than the same budget spent on a low-fertility language, so the model also sees comparatively less real content in the under-represented language for the same training cost, compounding a data disadvantage with a representational one.

What mitigates it

Every mitigation is a tradeoff, not a fix without cost. A larger, more inclusive tokeniser vocabulary can allocate more merge budget to non-English scripts, at the cost of a bigger embedding table or, for a fixed vocabulary size, fewer merges available per language overall. Training or fine-tuning language-specific or regional tokenisers is another route. So is fitting a whitespace-and-script-agnostic base, such as SentencePiece's Unigram model (see unigram-sentencepiece), on a genuinely balanced multilingual corpus from the start, rather than patching an English-centric vocabulary after the fact.

When it falls down

• Vocabulary size is a scarce, shared resource. A bigger shared vocabulary helps under-represented languages but dilutes the merge budget available to every language; there is no free lunch here.
• Per-word fertility is not directly comparable across languages. Languages differ enormously in average word length and writing conventions, agglutinative languages, or languages without whitespace-delimited words, so a per-byte reference unit is more defensible than per-word for cross-language comparison.
• Fixing fertility does not fix data scarcity. A language can have efficient tokenisation and still be poorly served by a model if the model saw little training text in that language to begin with; the two problems are related but distinct.