Anatomy of a Transformer Block

Open the weights file of any frontier LLM and you will find the same small function repeated 32, 80, or 120 times with independent numbers plugged in. Transformer Architecture tells you what the block is made of at a high level; this concept walks through what actually happens to a tensor as it passes through one, with shapes, so the block stops being a diagram and becomes arithmetic you can check.

The input and the two residual updates

A block receives a tensor x of shape [batch, seq_len, d_model], one vector per token, and returns a tensor of the identical shape. That shape-in-shape-out invariance is the entire trick that makes stacking possible: block 2 cannot tell whether block 1 is a transformer block or a no-op, so you can chain as many as you like. In the pre-norm form nearly every current model uses (see layernorm-residual-connections for why pre-norm won), the block does exactly two updates to the residual stream:

x = x + Attention(LN1(x))
x = x + FFN(LN2(x))

Each line reads: normalise a copy of the stream, transform it, add the result back to the un-normalised stream. The residual stream itself is never overwritten wholesale, only added to. Everything a block "knows" is expressed as a delta it contributes, not a replacement it imposes.

Sublayer one: attention, with shapes

LN1(x) still has shape [batch, seq_len, d_model]. Three learned projections, W_Q, W_K, W_V, each [d_model, d_model], produce Q, K, V of the same shape. These are reshaped and split into h heads, each of dimension d_k = d_model / h, so the model computes h independent attention operations in parallel rather than one large one. Each head computes softmax(Q K^T / sqrt(d_k)) V (see attention-mechanism for why the scale factor is there), producing an output of shape [batch, seq_len, d_k] per head. The heads are concatenated back to d_model and passed through one more projection, W_O, before the residual add. Four square matrices in total, Q, K, V, O, each d_model x d_model: roughly 4 * d_model^2 parameters for the whole attention sublayer, independent of sequence length.

Sublayer two: the feed-forward network

LN2(x) feeds a position-wise MLP that expands to a wider hidden dimension, conventionally 4 * d_model, applies a non-linearity, and projects back down (see feedforward-swiglu for the modern gated version). Two matrices, each roughly d_model x 4*d_model, put this sublayer at about 8 * d_model^2 parameters, twice the size of attention. That ratio is not a rounding error: in a typical block, the feed-forward sublayer holds about two-thirds of the parameters and attention holds about one-third, which is why FFN weights, not attention weights, dominate a model's memory footprint and are the primary target for quantisation and for mixture-of-experts sparsification.

What each sublayer is actually for

The two sublayers do structurally different jobs, and the difference explains why they run in this order and not some other. Attention is the only operation in the block that lets information move between token positions; every other operation, including the entire FFN, is applied to each position independently, with no knowledge of any other token. So attention mixes, and the FFN computes on what has been mixed. Running FFN before attention within a block would mean processing a token's representation before it has incorporated any information from its neighbours, wasting the FFN's capacity on an under-informed input. This ordering is not an arbitrary convention inherited from the original paper; it reflects a genuine dependency in what information is available when.

Compute, not just parameters

Parameter count and compute cost diverge as sequence length grows. Attention's cost scales as O(seq_len^2 * d_model), because every token compares itself against every other token; the FFN's cost scales as O(seq_len * d_model^2), linear in sequence length. For short-to-moderate sequences relative to d_model, the FFN dominates FLOPs, matching its parameter dominance. Push the sequence long enough and the quadratic attention term overtakes it, which is the whole reason long-context serving needs different engineering (see context-windows-long-context) than short-context serving does.

When it falls down

"One block, many copies" undersells what depth buys. The block's shape invariance says nothing about whether stacking more of them helps; that is a separate, genuinely open design question, see depth-width-aspect-ratio. Interpretability work also shows different depths specialise, early blocks pick up more local and syntactic patterns, later ones more abstract and task-specific ones, so "identical function repeated" is true of the architecture, not of what gets learned in each copy.
Treating the block as atomic hides real engineering choices. Whether attention and FFN run sequentially or from the same normalised input in parallel is itself a design decision with measurable throughput consequences, see parallel-attention-and-ffn. So is exactly where normalisation sits, see sublayer-ordering-design.
The clean two-line description omits the final norm. In a pre-norm stack, the residual stream itself is never normalised inside the loop, only sublayer inputs are, so an extra LayerNorm after the very last block is required before the output projection. Forgetting it is a common reimplementation bug.
Fixed shape hides variable cost per token. Every token pays the same FLOPs regardless of how "hard" it is to predict, a uniformity that later motivated conditional-compute designs like mixture-of-experts, which route different tokens to different subsets of FFN weights instead of running the same dense block on everything.