Sub-Layer Ordering and Design Choices

Layer normalization and residual connections covers the single biggest ordering decision in a transformer block, whether normalisation sits before or after the residual add. That is one design choice out of several. This concept surveys the rest of the menu: whether sublayers run sequentially or in parallel, where the stack's final normalisation goes, whether a block normalises both ends of a sublayer instead of just its input, and a handful of smaller conventions that shape how a block actually behaves.

Why attention always runs before the FFN

Sequential blocks put self-attention first and the feed-forward network second, and this is not an arbitrary convention. Attention is the only operation in a block that moves information between token positions; the FFN operates on each position independently, with no visibility into any other token (see anatomy-of-a-transformer-block). Running the FFN on a token before attention has let that token incorporate anything from its neighbours would waste the FFN's per-position computation on an under-informed input. Attention-then-FFN reflects a genuine information dependency within a single block, even though, across the full stack, the two interleave and repeat many times.

The parallel formulation used by GPT-J, GPT-NeoX, and PaLM (see parallel-attention-and-ffn) sidesteps this ordering question entirely by having both sublayers read the same unmixed input rather than one reading the other's output, trading the within-block composition for reduced synchronisation.

Where the final normalisation goes

Pre-norm architectures normalise the input to each sublayer but never normalise the residual stream itself inside the loop; the stream just accumulates additions, block after block. That means an extra LayerNorm, applied once, after the last block and before the output/unembedding projection, is required to bring the accumulated stream back to a scale the unembedding matrix expects. This final norm is easy to omit by accident when reimplementing a model from a paper, and its absence tends to produce a working-looking but subtly miscalibrated model, since the residual stream's scale can drift substantially over dozens of layers.

Sandwich norm and controlling residual growth

Plain pre-norm is x = x + Sublayer(LN(x)): normalisation gates what the sublayer reads, but nothing bounds what it writes back into the residual stream. Some architectures add a second normalisation after the sublayer and before the addition, a sandwich or double-norm pattern: x = x + LN_out(Sublayer(LN_in(x))). This directly bounds the magnitude of each block's contribution to the residual stream, not just its input, which addresses a documented failure mode of pure pre-norm at very high depth: residual-stream variance that grows large enough to destabilise training. Wang et al., 2022 (DeepNet) introduced explicit variance-controlling scaling in this spirit specifically to make 1,000-layer transformers trainable at all, well beyond the depths where plain pre-norm remains adequate on its own. Some recent open models adopted a related pattern, normalising both the input and output of each sublayer, for similar stability reasons at more modest depths.

Smaller choices in the same space

Whether Q, K, and V projections are implemented as one fused matrix or three separate ones is functionally identical but not equally efficient: a single fused projection is one larger matmul instead of three smaller ones, which most hardware executes faster. Whether attention and FFN projections include bias terms at all is another small but real choice; several recent models drop biases entirely, reporting negligible quality difference alongside a small saving in memory and compute. Neither choice changes what the model can represent, but both affect what it costs to run.

When it falls down

• Stability fixes validated at extreme depth can be pure overhead at moderate depth. Sandwich-norm and DeepNet-style variance scaling address failure modes that appear at hundreds or thousands of layers; applied to a 12- or 24-layer model, the extra normalisation is mostly cost with little corresponding benefit, so "more stabilisation is always better" does not hold.
• Sandwich norm doubles the per-sublayer normalisation cost, two norms instead of one, for a benefit that is scale-dependent; most production models still ship plain pre-norm because the simpler recipe is adequate at the depth they actually use.
• Ordering changes are rarely ablated in isolation. A paper that introduces a new sublayer ordering typically ships it alongside a new initialisation scheme, a new aspect ratio, and a new data mixture, all at once; crediting a quality improvement specifically to the ordering choice, separate from everything else that changed alongside it, is usually not fully justified by the evidence presented.
• The attention-before-FFN convention is about within-block dependency, not a law. The parallel formulation demonstrates the ordering is a genuine design choice, not a structural necessity, and that relaxing it costs something specific and measurable rather than nothing.

Further reading

• Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762 - the original post-norm, sequential ordering.
• Xiong et al., 2020, On Layer Normalization in the Transformer Architecture, arXiv:2002.04745 - the pre-norm versus post-norm analysis this concept builds on.
• Wang et al., 2022, DeepNet: Scaling Transformers to 1,000 Layers, arXiv:2203.00555 - variance-controlling normalisation to train extremely deep stacks.