NLP Foundations
The Prompt Stack and Chat Roles
What a chat prompt actually is under the hood: a single token sequence built from system, user, and assistant turns wrapped in special tokens, and why that structure is load-bearing.
intermediate · 6 min read
A chat with an assistant looks like a conversation between distinct parties. Under the hood it is one flat sequence of tokens fed to an autoregressive model, with special tokens marking where each "turn" begins and who is speaking. There is no separate memory and no separate channel per speaker; the roles are a convention rendered into the token stream.
The three roles
- System: the standing instructions, persona, and constraints. It sits first and conditions everything after it.
- User: the human's messages.
- Assistant: the model's own prior replies, fed back in so it can see what it already said.
A chat template stitches these into a single string with delimiter tokens, for example a special token that means "start of a system message" and another that means "start of assistant turn". The model was fine-tuned on exactly this format, so getting the template wrong (missing a delimiter, wrong role order) can degrade output badly even with a correct-looking prompt.
Everything is in the sequence
The single most important consequence: the model's entire knowledge of the conversation is whatever is currently in that token sequence. It has no hidden recollection of turn 1 once turn 1 scrolls out of the window. "Memory" in a chat product is an illusion maintained by re-sending the history on every call. This is why the sequence is state, and why managing it is context engineering.
The prompt stack as a budget
Line up the pieces and a prompt is a stack competing for a finite token budget:
[ system instructions ][ tool definitions ][ retrieved data ][ message history ][ current user turn ]
Every element is tokens the model pays attention over on every single call. Tool definitions that are never used, stale history, and over-long system prompts all consume budget and can trigger context rot. Seeing the prompt as a layered stack, rather than "the thing I typed", is the first step toward controlling agent behaviour.
Why roles matter for safety and control
Role separation is also a control surface. Instructions in the system role are meant to outrank content in the user role, which is the mechanism behind guardrails. It is an imperfect boundary: prompt injection works precisely by smuggling instructions into user-role or retrieved content and getting the model to follow them as if they were system-level. Understanding the stack is what lets you reason about where an instruction actually sits and how much authority it should carry.