Why Non-Linearity Matters

Here is a fact that surprises people meeting deep learning for the first time: a neural network built entirely from linear layers, no matter how many, is mathematically equivalent to a single linear layer. A hundred-layer network with no non-linearity collapses to one matrix multiplication. Every claim about deep networks learning "hierarchical representations" or "complex functions" secretly depends on a small, cheap operation inserted between the linear layers: the activation function.

The collapse, shown directly

If y = W2 * (W1 * x), that is the same as y = (W2 * W1) * x, and W2 * W1 is just another matrix. Composing linear (technically affine) transformations always produces another linear transformation; matrix multiplication is associative, so the composition can always be pre-multiplied into a single matrix. Depth without non-linearity buys nothing beyond what a single, appropriately sized linear layer already provides, and in fact costs more compute to achieve the same expressive power (see matmul-the-core-operation).

linear(linear(x)) = W2 (W1 x) = (W2 W1) x = one_linear_layer(x)

What a non-linearity buys

Insert a non-linear function f between the two linear layers, y = W2 * f(W1 * x), and the collapse no longer happens; f cannot be absorbed into a matrix. This is what gives a network the capacity to approximate functions that are not themselves linear, formalised by the universal approximation theorem: a network with even one hidden layer and a suitable non-linearity can approximate any continuous function on a bounded domain to arbitrary precision, given enough width. That theorem says nothing about how easy such a function is to learn or how many parameters it takes in practice, which is exactly why depth (rather than pure width) turned out to matter empirically: a deep, narrow non-linear network can represent compositions of simple functions far more parameter-efficiently than a single wide layer.

ReLU, GELU, and the practical choices

The specific shape of the non-linearity has mattered enormously in practice, mostly for how well gradients flow through it (see the-backward-pass-gradient-flow), not for the universal-approximation argument, which holds for a wide family of functions. ReLU (max(0, x)) is cheap and has a clean gradient (1 or 0), but that "0" is also its weakness: a unit whose input is always negative contributes zero gradient forever, a dead unit. GELU and Swish smooth out that hard cutoff, giving a small negative-region gradient, which is why they largely replaced plain ReLU in transformer feed-forward blocks (see activation-functions and feedforward-swiglu for the gated variant used in most current models).

Where the non-linearity actually lives in a transformer

It is easy to assume attention is the "non-linear" part of a transformer, since softmax is technically non-linear, but the feed-forward block's activation function is the workhorse non-linearity in terms of representational capacity, applied independently to every position, expanding then contracting the hidden width around it. Attention is closer to a content-dependent linear mixing of value vectors; the genuine function-approximation power leans heavily on the position-wise feed-forward non-linearity, applied identically at every position, layer after layer.

When it falls down

• Not all non-linearity is equally useful. A non-linearity that saturates over most of its input range (like sigmoid, in its tails) contributes little gradient signal even though it is technically non-linear, which is why the choice of activation shape, not just its non-linearity, is an active design decision.
• Universal approximation is not a training guarantee. The theorem says a sufficiently wide network with non-linearity can represent a target function; it says nothing about whether gradient descent will find the parameters that do so, or how much data that would take.
• Piecewise-linear activations are still globally non-linear. ReLU is linear on each side of zero, which sometimes leads to the mistaken intuition that it is "basically linear"; the kink at zero is precisely what prevents the multi-layer collapse described above.
• Too much non-linearity has costs too. Non-linear functions are exactly where gradients can saturate or die (see the dead-ReLU problem above), so the same property that gives representational power also introduces the specific failure modes that initialisation and activation-function research spend most of their effort mitigating.

Further reading

• Hornik, Stinchcombe, White, 1989, Multilayer Feedforward Networks are Universal Approximators - the original universal approximation theorem.
• Hendrycks and Gimpel, 2016, Gaussian Error Linear Units (GELUs), arXiv:1606.08415 - the smoothed activation now standard in most transformer feed-forward blocks.