NLP Foundations
Tokenisation and Arithmetic
A model can be strong at multi-step reasoning and still botch four-digit addition, not from a lack of arithmetic ability but because its tokeniser handed it two structurally unrelated token sequences for two numbers that differ by one.
intermediate · 7 min read
A model can be extremely good at multi-step reasoning and still botch four-digit addition. Not from a lack of "math ability," but because its tokeniser handed it "1234" and "1235" as two structurally unrelated token sequences instead of two numbers that differ by one in the last place.
The place-value problem
Standard place-value arithmetic, the kind taught in school and the kind a transformer would need an analogous internal circuit for, depends on digits being aligned by position: ones under ones, tens under tens. A frequency-driven BPE vocabulary has no concept of place value. It merges whatever digit sequences are common in its training corpus, so a common year like "1990" might collapse into a single token while a superficially similar four-digit number splits as two digits and two digits, or as one digit and three. That inconsistency means the same arithmetic algorithm the model might otherwise learn cannot be applied uniformly, because the input representation of "the same kind of number" is not itself uniform.
What later systems changed
Newer tokeniser configurations, adopted broadly across the field from the GPT-3.5 and GPT-4 era onward, constrain numeral tokenisation deliberately, most commonly by tokenising digits individually or in fixed-size groups, such as always splitting a number into chunks of up to three digits counted from the right, rather than letting frequency-driven merges decide. This does not teach the model arithmetic by itself, but it removes a representational obstacle: with consistent, position-aligned digit grouping, the same learned sub-circuit can generalise across numbers instead of relearning ad hoc patterns for every digit grouping the frequency-based vocabulary happened to produce. Nogueira, Jiang, and Lin, 2021, studying transformers on simple arithmetic tasks, found tokenisation and digit ordering to be major, controllable levers on performance, separate from raw model scale.
Why this generalises beyond addition
The same fragility shows up anywhere an algorithm needs consistent structure across tokens: sorting numbers, comparing magnitudes, counting, unit conversions involving many digits. It is a sibling problem to why-llms-cant-spell: both are cases where a task needs access to structure below the token boundary, and the tokeniser's job, compress common patterns, is directly in tension with the model's need for uniform, decomposable structure on these specific tasks.
When it falls down
- Digit-level tokenisation is not free. It costs more tokens per number, a direct instance of the-tokenisation-tax; numeric-heavy workloads, financial data, scientific notation, pay proportionally more.
- It helps addition and comparison more than multiplication. Multiplying large numbers still requires the model to implicitly learn something like long multiplication over many steps, a separate capability limitation beyond tokenisation alone.
- Tool use sidesteps the problem entirely. Most production systems rely on an actual calculator call for anything beyond small arithmetic; tokenisation fixes are a partial mitigation, not a substitute for that.
Further reading
- Nogueira, Jiang, Lin, 2021, Investigating the Limitations of Transformers with Simple Arithmetic Tasks - controlled experiments isolating tokenisation and digit ordering as performance levers.
- Sennrich, Haddow, Birch, 2016, Neural Machine Translation of Rare Words with Subword Units - the frequency-driven merge process that produces inconsistent digit grouping in the first place.