NLP Foundations
Byte-Level BPE
Why GPT-2 and its descendants tokenise raw bytes instead of characters, and how that single design choice guarantees any input string can be represented without an unknown-token fallback.
intermediate · 7 min read
GPT-2's vocabulary has exactly 256 base symbols before a single merge happens, one for every possible byte value. That is the whole reason it never needs an unknown-token fallback: any string, in any language, with any emoji or malformed encoding thrown in, is already representable as a sequence of raw bytes before the tokeniser does anything clever at all.
From characters to bytes
The original byte-pair encoding algorithm, described by Sennrich et al., 2016 for neural machine translation, operates on Unicode characters, not bytes. That means the base alphabet has to include every character type the training corpus contains, and it grows quickly once you cover more than one script: Latin letters, digits, punctuation, and then, if you want coverage, every character in Chinese, Japanese, Korean, Devanagari, Cyrillic, Arabic, and so on. Any character absent from that seed alphabet at inference time is unencodable and falls back to an unknown-token, a lossy, information-destroying failure mode.
Byte-level BPE, used by GPT-2, reframes the problem entirely. Treat the input as its raw UTF-8 byte stream rather than as characters. The base alphabet is then fixed at exactly 256 symbols, one per byte value, regardless of which language, script, or encoding produced the text. Every possible byte sequence, including ones that are not even valid UTF-8, is representable from the base alphabet alone, before any merge has been learned (see tokenisation-bpe for the merge mechanics themselves).
The reversible remapping trick
GPT-2, described in Radford et al., 2019, does not run BPE directly on raw byte values. Bytes 0 through 31 and several others are control characters or whitespace that behave badly inside standard text-processing and regex-based pre-tokenisation pipelines. Instead, GPT-2 remaps the 256 byte values one-to-one onto printable Unicode code points, a reversible bijection, and runs the BPE merge algorithm over these stand-in symbols. Decoding simply inverts the mapping. The net effect is a tokeniser that manipulates human-readable-ish symbols during training and inference while remaining byte-exact and fully reversible: encode then decode always recovers the original bytes precisely.
Why fewer base symbols does not mean fewer tokens
Coverage and efficiency are different properties, and byte-level BPE only guarantees the first. Merges are learned by frequency over the training corpus, so a script that appeared rarely, or not at all, in that corpus accumulates few useful merges. A single Devanagari or Han character, which takes two to four bytes in UTF-8, starts as multiple base tokens and needs several learned merges to collapse into one or two tokens the way a common English word does. When the tokeniser's training data under-represents a script, those merges simply never get learned, and the script stays fragmented at something close to its raw byte-level length. This is the mechanical root of the disparity explored in the-tokenisation-tax and, at the level of measured cross-language cost, in token-fertility-multilingual.
When it falls down
- Coverage is not efficiency. Guaranteeing no unknown-token fallback says nothing about how many tokens a given string costs; a script under-represented in the tokeniser's training corpus can still fragment heavily.
- Greedy merge order is linguistically arbitrary. Merges are chosen purely by frequency, so word boundaries in the resulting vocabulary do not necessarily track morpheme or syllable boundaries, unlike the likelihood-driven selection in wordpiece-tokenisation or the probabilistic model in unigram-sentencepiece.
- Segmentation is deterministic. The same string always tokenises identically at inference; introducing variation requires a deliberate technique such as bpe-dropout-subword-regularisation.
- Multi-byte characters can split across tokens, which matters directly for anything that decodes output incrementally (see detokenisation-and-streaming).
Further reading
- Sennrich, Haddow, Birch, 2016, Neural Machine Translation of Rare Words with Subword Units - the original BPE algorithm, applied at the character level.
- Radford et al., 2019, Language Models are Unsupervised Multitask Learners (GPT-2) - introduces the byte-level extension and the reversible byte-to-unicode mapping.