Signal Propagation and Initialisation

Set every weight in a deep network to the same small constant and it will not train, not because the optimiser is bad, but because the forward pass itself destroys the signal before training even begins. Initialisation is the most consequential decision made before the first gradient step, and it is not a minor implementation detail: it decides whether variance grows, shrinks, or stays roughly constant as activations pass through dozens of layers.

Why variance is the quantity that matters

Consider a single linear layer y = Wx with x having d independent components of variance var(x), and W's entries drawn independently with variance var(W). Each output entry is a sum of d products, and for independent, zero-mean terms, variances add:

var(y) = d * var(W) * var(x)

If var(W) is chosen so that d * var(W) = 1, the layer preserves input variance exactly: var(y) = var(x). This is the entire idea behind principled initialisation schemes: choose the weight variance as a function of layer width so that signal neither shrinks toward zero (vanishing) nor grows without bound (exploding) as it passes through many layers, before any learning has happened. A network initialised badly can have activations that are already saturated or already zero at layer 10, in which case gradients computed via the chain rule (see the-backward-pass-gradient-flow) have almost nothing to propagate.

Xavier/Glorot initialisation

Glorot and Bengio, 2010 derived a variance that balances both the forward pass (preserving activation variance) and the backward pass (preserving gradient variance), for networks using symmetric activations like tanh:

var(W) = 2 / (fan_in + fan_out)

Averaging fan_in and fan_out is a compromise: exactly matching the forward-pass condition alone would use 1/fan_in, and exactly matching the backward-pass condition alone would use 1/fan_out; Xavier splits the difference so neither direction is badly served.

Kaiming/He initialisation

Xavier assumes a roughly linear, symmetric activation. ReLU breaks that assumption: it zeros out half its input on average, which halves the variance the Xavier formula was designed to preserve. He et al., 2015 corrected for this by accounting for ReLU's variance-halving effect:

var(W) = 2 / fan_in

The factor of 2 exactly compensates for the fact that a ReLU unit's output has, on average, half the variance of its input (since roughly half the values are clipped to zero). This is the standard initialisation for networks with ReLU-family activations, and it was originally motivated by making very deep convolutional networks trainable from scratch without auxiliary tricks.

Why transformers still need help beyond a good initial variance

Even with correct per-layer variance, a residual stream x = x + Sublayer(x) accumulates variance across depth, because each block's output adds new variance to the running sum rather than replacing it. Left unaddressed, the residual stream's variance grows roughly linearly with depth, which is why some architectures apply an additional depth-dependent scaling factor to sublayer outputs (scaling residual contributions down as depth increases) so that a 100-layer model's final activations are not orders of magnitude larger in scale than a 10-layer model's. Pre-norm placement (see layernorm-residual-connections) mitigates but does not fully eliminate this growth, since normalisation rescales the input to each sublayer without rescaling the accumulating residual stream itself.