The Residual Stream

A transformer block computes y = x + Sublayer(x) (see layernorm-residual-connections). Stated that way it sounds like plumbing. Stated differently, it is the entire architecture: every attention head and every feed-forward layer in the network reads from and writes to one shared, persistent vector of width d_model, and that vector is the only thing that flows from the embedding layer at the bottom to the unembedding at the top. Elhage et al., 2021 named this the residual stream and showed that thinking of it as a single object, rather than a sequence of separate layer outputs, resolves a lot of what otherwise looks mysterious about how transformers compute.

Why the additive structure matters

Because every sublayer's contribution is added to the stream rather than replacing it, the residual stream at the final position can be decomposed as a literal sum: the initial embedding, plus every attention head's output, plus every MLP's output, each one an independent additive term. Nothing except the layernorm rescaling before each sublayer nonlinearly mixes these contributions together. This is why the logit lens works at all: since the space never changes basis, only accumulates, you can take the partial sum at an intermediate layer and read it through the final unembedding matrix and get a meaningful, if incomplete, answer, because it is the same coordinate system the final read was trained on (helped further when embedding and unembedding are tied, see weight-tying-embeddings).

Each attention head or MLP neuron, in this framing, is a small function: it reads a linear projection of the current residual stream (its "input subspace") and writes an additive update into another linear projection (its "output subspace"). Two components can read overlapping information or write to overlapping directions without either being aware of the other, they only interact through what has already accumulated in the shared stream.

The bottleneck this creates

d_model is fixed, typically a few thousand dimensions, but a model needs to represent far more distinct things than that: thousands of factual associations, syntactic roles, entity identities, task state, and more. The residual stream cannot give every one of these its own dedicated orthogonal dimension; there simply is not room. This scarcity is the direct motivation for superposition, where the model represents more features than it has dimensions by using directions that are only approximately, not exactly, orthogonal (see superposition-and-polysemantic-neurons). The residual stream is, in effect, always somewhat crowded, and everything downstream, including how cleanly interpretable a given direction is, follows from that scarcity.

When it falls down

• Bandwidth is shared and finite, so components interfere. Every head and MLP competes for the same limited set of "slots" in the stream, which is part of why ablating a single attention head can produce surprising, seemingly unrelated downstream effects, another component may have been relying on the same subspace.
• Layernorm is a real nonlinearity, not a footnote. The clean additive-decomposition story assumes near-linearity, but the layernorm applied before every sublayer rescales based on the whole current vector, so the "sum of independent contributions" picture is an approximation, not an exact one.
• Directions found by probing are not automatically causal. A linear probe finding a "sentiment direction" or "truthfulness direction" in the residual stream shows correlation; showing the model actually uses that direction for its output requires a causal intervention (such as activation patching), not just a probe.
• This is an interpretability abstraction, not a design blueprint. Nobody explicitly built transformers around the residual-stream framing; it is a useful lens applied after the fact, and treating it as literally how the architecture was engineered overstates its status.