BPE-Dropout and Subword Regularisation

A deterministic tokeniser has a blind spot that no amount of extra training data fixes on its own: the model only ever sees each word segmented one specific way, so it never has to be robust to being wrong about where a word boundary falls, right up until inference hands it a segmentation it has never trained on.

The precursor: subword regularisation

Kudo, 2018 introduced subword regularisation alongside the Unigram LM tokeniser (see unigram-sentencepiece). Because the unigram model assigns a probability to every valid segmentation of a string, training can sample a different, still-valid segmentation of the same word on different training examples instead of always using the single best (Viterbi) one, implemented via a forward-filtering, backward-sampling procedure over the segmentation lattice. The model then has to learn to produce the right output regardless of which valid subword split it happened to be handed, which empirically improved translation quality and robustness, especially on rare and compound words.

BPE-Dropout

Provilkov, Emelianenko, Voita, 2019 brought the same idea to plain BPE, which has no natural notion of segmentation probability to sample from in the first place. The mechanism: at tokenisation time, randomly drop each learned merge with some probability p, typically a small value such as 0.1, so the encoder sometimes skips a merge it would otherwise apply and falls back to a less-merged, more granular segmentation for that instance. Set p = 0 at inference and BPE-Dropout is a strict no-op, identical to standard BPE. It changes nothing about the vocabulary, the architecture, or inference-time cost; it is purely a training-time augmentation.

Why it helps

Two related effects are at work. First, it is a data augmentation that multiplies the effective number of distinct token sequences the model sees for the same underlying text, without adding a single byte of new raw data. Second, it hardens the model against exactly the class of tokenisation instability discussed in the-tokenisation-tax and detokenisation-and-streaming: the same text can be split differently depending on adjacent context, casing, or minor encoding differences, and a model trained under subword regularisation degrades more gracefully when it meets an unfamiliar segmentation of a familiar word at inference, rather than treating the input as entirely novel.

When it falls down

• Training-time only. It does nothing for a model already pretrained with deterministic tokenisation; you cannot retrofit the robustness after the fact without further training on regularised data.
• A hyperparameter tradeoff. The dropout probability trades regularisation strength against training-signal quality; set too high, the model spends capacity learning to parse degenerate segmentations rather than the underlying task.
• Does not fix the vocabulary itself. It makes the model more robust to which valid segmentation of already-representable text it receives, but it does nothing for coverage or fairness problems baked into the base vocabulary (see token-fertility-multilingual).