Detokenisation and Streaming Boundaries

Stream a multilingual chat response character by character and, every so often, the screen briefly shows a garbled replacement-character glyph before self-correcting a moment later. This is not a UI bug. It is a direct, visible consequence of a single emoji or non-Latin character being split across more than one token.

Why a "character" and a "token" are different units

A tokeniser's vocabulary is built over byte or subword units, with no guarantee of one-to-one alignment with a displayed character. A multi-byte UTF-8 character, accented Latin letters, most non-Latin scripts, and many emoji, which can take up to four bytes, can require two or more tokens to represent completely (see byte-level-bpe). When a model streams output one token at a time, printing each token's decoded bytes to the screen immediately, a token representing only the first half of a multi-byte character produces an invalid, incomplete byte sequence on its own. Naively decoding that partial sequence either throws an error or renders a Unicode replacement character until the remaining token arrives.

The correct pattern: buffer to a valid boundary

Robust streaming detokenisers do not decode and flush every token in isolation. They maintain a small byte buffer, append each new token's bytes to it, and attempt to decode the buffer as UTF-8, flushing and displaying only the characters that are already complete, while keeping any trailing incomplete byte sequence in the buffer for the next token. This is the same incremental-decoder pattern used for any streaming byte source, a network socket reading UTF-8 text, applied at the token boundary instead of an arbitrary byte boundary.

SentencePiece adds a second kind of boundary problem

Beyond byte-completeness, SentencePiece-style tokenisers (see unigram-sentencepiece) encode whitespace explicitly via a leading marker such as ▁. A correct detokeniser must track whether it has already emitted the space for the current word, so it does not duplicate or drop a space when a token stream is interrupted or resumed mid-word. This is easy to get right for a whole-sequence decode and easy to get wrong for a token-by-token streaming decode, since the marker's meaning depends on decode state, not just the current token in isolation.

When it falls down

• Truncated streams can strand a partial character. A timeout or cancelled request can leave the client holding an incomplete multi-byte character indefinitely if the buffering logic has no final flush-or-discard step on stream close.
• UI assumptions break on multi-token characters. Anything that assumes "one token equals one visible unit," a progress bar, a typing-speed animation, will visibly stutter whenever a multi-token character streams in; this is a real UX detail, not just a correctness one.
• Byte-fallback tokens need special handling. Some tokenisers include explicit byte-fallback tokens for invalid or unmappable sequences, and treating those as ordinary text tokens instead of raw-byte markers reintroduces the same corruption the buffering pattern was meant to prevent.

Further reading

• Kudo and Richardson, 2018, SentencePiece - the explicit whitespace-as-symbol convention that streaming decoders must respect.
• Radford et al., 2019, Language Models are Unsupervised Multitask Learners (GPT-2) - the byte-level base alphabet that makes multi-byte splits possible in the first place.