WordPiece Tokenisation

BERT's vocabulary looks superficially like BPE's: subword pieces, a marker for pieces that continue a word rather than start one. But the algorithm that built it optimises a genuinely different objective, and that difference is why WordPiece pieces tend to look more like actual morphemes than an arbitrary frequency-driven split.

The merge criterion: likelihood, not frequency

Plain BPE merges whichever adjacent pair is most frequent in the corpus (see tokenisation-bpe). WordPiece instead merges the pair that most increases the likelihood of the training corpus under a unigram language model built from the current vocabulary. Roughly:

score(a, b) = count(a, b) / (count(a) * count(b))

This is a mutual-information-style measure: it rewards pairs that co-occur more often than their individual frequencies would predict, and penalises pairs that are each individually common but only coincidentally adjacent. A raw-frequency merge criterion (plain BPE) can be fooled by two very common symbols that happen to sit next to each other often simply because both are common everywhere; WordPiece's ratio resists that.

Continuation marking with

GPT-2-style byte-level BPE marks a word's first token with a leading-space marker. WordPiece marks the opposite: a continuation piece is prefixed with ##, so "playing" becomes play + ##ing, meaning "glue me to the token on my left, no space in between." The convention makes detokenisation mechanical (strip the space before any ## token, then drop the marker) and it lets you inspect a token sequence and immediately see where word boundaries fall, which matters for BERT-style downstream tasks like named-entity recognition and span tagging that operate over whole words rather than raw token spans.

Encoding is still greedy longest-match

Despite the different merge selection at training time, encoding new text with a finished WordPiece vocabulary is greedy longest-match-first, the same algorithmic family as BPE. For a given word, the encoder repeatedly finds the longest vocabulary entry that matches the remaining unconsumed prefix, emits it, and continues on what is left, marking continuations with ##. If no valid prefix exists at some point, the entire word falls back to [UNK], a real limitation that byte-level BPE's fixed 256-byte base alphabet was specifically designed to avoid (see byte-level-bpe): a classic WordPiece vocabulary is typically built over a Unicode character seed set, not raw bytes, so a genuinely novel character can still force the fallback.

WordPiece was introduced by Schuster and Nakajima in 2012 for a Japanese and Korean voice search system, and it became widely known through Devlin et al., 2018, BERT, which used it for the model's [CLS], [SEP], and subword vocabulary throughout.

When it falls down

• Word-level unknown-token risk. A character-based seed alphabet, unlike byte-level BPE's guaranteed byte coverage, still admits [UNK] for characters the training corpus never saw.
• More expensive to train. The likelihood-based merge score requires re-estimating a unigram model at each step, costlier than BPE's simple frequency count, which is one reason later byte-level systems favoured plain BPE for training-time cost.
• Assumes a pre-tokenisation step. ## marking presupposes text has already been split on whitespace and punctuation, so WordPiece does not by itself give you the whitespace-agnostic segmentation that unigram-sentencepiece provides for languages without spaces between words.

Further reading

• Devlin, Chang, Lee, Toutanova, 2018, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - the model that popularised WordPiece.
• Schuster and Nakajima, 2012, Japanese and Korean Voice Search, IEEE ICASSP - the original WordPiece algorithm, predating widespread arXiv preprint conventions.