Special and Control Tokens

Somewhere in every large language model's vocabulary sits a handful of tokens no amount of ordinary BPE merging could ever produce from user-supplied text. Their entire job is to tell the model "this is not content, this is structure," and the mechanics of keeping that boundary intact matter more than the bookkeeping might suggest.

What they mark

• BOS and EOS. Beginning and end of sequence. EOS often doubles as the stop signal during generation: the model is trained to emit it when it considers a response complete, and the decoding loop halts once that token is sampled (see autoregressive-generation).
• PAD. Fills a batch out to equal length so sequences can be processed together; the model is trained or masked to ignore it entirely.
• UNK. The fallback for anything the vocabulary genuinely cannot represent. Rare under byte-level BPE (see byte-level-bpe), more common under a classic character-seeded WordPiece vocabulary.
• Role and turn tokens. Chat-tuned models add tokens marking system, user, and assistant message boundaries, so the model can distinguish instructions from content within a single flattened token stream.
• Tool and function-call delimiters. Agentic models add tokens marking the start and end of a tool invocation and its arguments, kept separate from a free-text response.
• Fill-in-the-middle tokens. Code models add prefix, suffix, and middle markers so the model can be trained to complete a gap given both the code before and after it, not just what precedes it.

Why they must sit outside the mergeable vocabulary

Special tokens are added after BPE, WordPiece, or Unigram training finishes, assigned reserved IDs, and the tokeniser's encoder is built to recognise their literal string form, such as <|endoftext|>, as an atomic unit before ordinary merge rules ever run on that span of text. The ordering matters. If a special token's surface string could instead be produced by ordinary subword merges over arbitrary user text, a user could type that literal string and have it parsed as the real control token, letting them inject a fake turn boundary or a fake system message into a prompt that was built by naive string concatenation. This is a real, recurring class of prompt-injection and tokeniser-confusion risk, not a theoretical concern, and it is why production tokenisers escape or specially handle any user-supplied text that happens to match a special token's surface form before encoding.

When it falls down

• Wasted embedding rows. Many vocabularies ship dozens of unused reserved special-token slots for forward compatibility, which quietly costs embedding table capacity.
• No shared convention across families. Different tokenisers mark the same concept differently, EOS versus <|endoftext|> versus </s>, so code that moves prompts between model families by string substitution alone is fragile.
• Token-boundary protection is not semantic protection. Correct special-token handling stops the literal injection attack, but it does not stop a model from being talked into treating ordinary user text as an instruction at the semantic level; tokenisation only defends the token boundary, not the model's interpretation of what is inside it.

Further reading

• Radford et al., 2019, Language Models are Unsupervised Multitask Learners (GPT-2) - introduces the <|endoftext|> convention.
• Devlin, Chang, Lee, Toutanova, 2018, BERT - [CLS] and [SEP] as structural, non-mergeable tokens.