Layer Normalization and Residual Connections

A transformer block has two headline components everyone talks about, attention and the feed-forward network, and two pieces of plumbing nobody notices until they are missing: the residual connection and the normalization layer. Remove either and a deep transformer simply will not train. They are the reason you can stack 80 or 120 blocks without the signal exploding or vanishing on the way through.

Residual connections: a highway for the gradient

A residual connection adds a sublayer's input back to its output: y = x + Sublayer(x). The point is what happens to the gradient. During backpropagation, the + x term creates a path where the gradient flows straight through, unmultiplied, so even if Sublayer contributes almost nothing early in training, the gradient still reaches the layers below undiminished. Without residuals, the gradient must pass through every transformation in sequence, and the product of many small Jacobians drives it toward zero (the vanishing-gradient problem that capped network depth before residual networks). Residuals turn a deep stack into an ensemble of shorter paths, and the model learns to use the deeper ones gradually.

Layer normalization: keeping activations in range

The other failure mode of deep networks is activations whose scale drifts layer to layer until they overflow or saturate. Layer normalization fixes this per token: for each position's activation vector, subtract its mean and divide by its standard deviation, then apply a learned scale and shift.

LN(x) = gamma * (x - mean(x)) / sqrt(var(x) + eps) + beta

Unlike batch normalization, LayerNorm normalises across the feature dimension of a single example, so it does not depend on other examples in the batch. That independence is why it suits sequence models: the statistics are stable whether the batch has one sequence or a thousand, and there is no train/inference discrepancy from batch statistics.

Pre-norm versus post-norm

The original transformer put the normalization after the residual add: x + Sublayer(x), then normalise (post-norm). It worked, but only with careful learning-rate warmup, because early in training the un-normalised residual stream could grow large and destabilise the update. Modern transformers moved the norm inside the residual branch, normalising the input to each sublayer: x + Sublayer(LN(x)) (pre-norm). This keeps a clean, un-normalised residual highway from input to output, so gradients flow even more directly, and it removes the need for delicate warmup. Nearly every model since GPT-2 uses pre-norm; the analysis of why is in Xiong et al., 2020.

RMSNorm: dropping the mean

LayerNorm does two things, re-centre (subtract mean) and re-scale (divide by standard deviation). RMSNorm (Zhang and Sennrich, 2019) asked whether the re-centring earns its cost and concluded it mostly does not. It normalises by the root-mean-square of the activation alone, with no mean subtraction and no bias term:

RMSNorm(x) = gamma * x / sqrt(mean(x^2) + eps)

Fewer operations, one fewer reduction, and empirically no quality loss. Llama, Mistral, Qwen, and most current open models use RMSNorm rather than full LayerNorm. It is a small change that compounds across every layer of every forward pass.

When it falls down

• Norm placement interacts with residual scaling. Very deep pre-norm models can see the residual stream's variance grow with depth, which is why some architectures add scaling factors or extra norms; naive stacking to hundreds of layers is not automatically stable.
• The epsilon matters. Too small an eps and low-variance activations blow up; too large and the norm barely normalises. It is a real hyperparameter, not boilerplate.
• Quantisation sensitivity. Normalization layers concentrate dynamic range, so they are often kept in higher precision even when the rest of the model is quantised (see low-bit quantization if available).

Further reading

• Ba, Kiros, Hinton, 2016, Layer Normalization, arXiv:1607.06450 - the original LayerNorm.
• He et al., 2015, Deep Residual Learning, arXiv:1512.03385 - residual connections, from vision but the idea transformers inherited.
• Xiong et al., 2020, On Layer Normalization in the Transformer Architecture, arXiv:2002.04745 - the pre-norm versus post-norm analysis.
• Zhang and Sennrich, 2019, Root Mean Square Layer Normalization, arXiv:1910.07467 - RMSNorm.