The Tokenisation Tax

"How are you?" in English and its equivalent in another language can tokenise into a different number of pieces under the very same vocabulary. That difference is not cosmetic. It compounds into real dollars, real context budget, and real latency, an invisible tax charged before the model has done any reasoning at all.

Where the tax gets paid

• API cost. Commercial pricing is per token, both input and output, so a string that tokenises into more pieces costs proportionally more, independent of how much meaning it actually carries (see tokenisation-bpe and token-fertility-multilingual).
• Context budget. A fixed context window, say 128k, is a fixed budget of tokens, not characters or words, so a verbose tokenisation of the same content eats through working memory faster (see context-windows-long-context).
• Decode latency. Generation is autoregressive: in the naive case, each output token costs its own sequential forward pass, so more output tokens for equivalent content directly means more sequential passes and higher wall-clock latency, compounded by a growing key-value cache that must be read at every step (see autoregressive-generation and kv-cache).

What drives the exchange rate

English prose under a general-purpose byte-level BPE tokeniser typically compresses to roughly 4 characters per token. Code, dense numeric text, and languages under-represented in the tokeniser's training corpus compress far worse, sometimes close to one token per character or worse for scripts that require several UTF-8 bytes per character (see byte-level-bpe). Formatting matters too: JSON with heavy punctuation and repeated key names tokenises worse per unit of information than equivalent prose, and unusual whitespace or indentation patterns in code can fragment into extra tokens depending on how the tokeniser handles whitespace.

Paying it down

Every mitigation is an engineering tradeoff, not a free lunch. Choose a tokeniser trained on a corpus that matches your traffic's actual language and domain mix. Strip boilerplate and redundant whitespace before sending text to the API. Prefer terser output schemas over verbose ones when the application can tolerate it. Where a workload is genuinely dominated by one language or domain, evaluate whether a specialised tokeniser, or a specialised model altogether, beats a general-purpose one on effective cost per unit of useful content.

When it falls down

• Token count is not a perfect proxy for information. Two prompts with identical token counts can carry very different amounts of task-relevant content, so chasing fewer tokens can trade away clarity.
• The tax is asymmetric and mostly invisible to whoever chooses the model. A developer testing prompts in English routinely under-estimates the real cost their non-English users will pay (see token-fertility-multilingual for the fairness argument in full).
• Fewer tokens is not always better for capability. Techniques like chain-of-thought prompting deliberately spend tokens to buy accuracy; the tax framing applies to unnecessary tokens, not to every token equally (see chain-of-thought).

Further reading

• Sennrich, Haddow, Birch, 2016, Neural Machine Translation of Rare Words with Subword Units - the compression mechanism underlying the cost differential.
• Kudo and Richardson, 2018, SentencePiece - a tokeniser designed explicitly for multilingual balance.