Feed-Forward Networks and SwiGLU

Attention gets the attention, but in a standard transformer block roughly two-thirds of the parameters sit in the other sublayer: the feed-forward network (FFN). It is applied independently to every position, it is where the model does most of its per-token computation, and a growing body of interpretability work suggests it is where a lot of the model's factual knowledge is actually stored. Understanding it is understanding where the weights, and the memory, really are.

The basic shape

The classic FFN is two linear layers with a non-linearity between them:

FFN(x) = W_2 * activation(W_1 * x + b_1) + b_2

The first layer projects the model dimension d up to a hidden dimension, conventionally 4 * d, the activation adds non-linearity, and the second layer projects back down to d. That 4x expansion is why the FFN dominates the parameter count: for d = 4096, the two matrices are each roughly 4096 x 16384, far larger than the attention projections. The FFN runs on each token separately with no mixing across positions; attention does the mixing, the FFN does the thinking.

Why a wide middle

The up-projection gives the layer room to compute many intermediate features in parallel before compressing back down. A useful lens from interpretability: the first matrix's rows act like keys that detect patterns in the input, the activation gates which fire, and the second matrix's columns act like values written into the residual stream. Under this view the FFN is a giant key-value memory, and specific facts can be localised to specific hidden units, which is what makes targeted model editing possible.

From ReLU to GELU to SwiGLU

The activation function evolved. The original transformer used ReLU (max(0, x)), simple but with a hard zero that discards information and a dead-gradient region. GELU (Hendrycks and Gimpel, 2016) replaced the hard cutoff with a smooth gate that weights each input by roughly the probability a Gaussian is below it, so small negative values pass through slightly rather than being zeroed. GELU became the default for BERT and GPT-family models.

The current default is a gated variant. GLU (gated linear unit) splits the up-projection into two halves and uses one to gate the other elementwise. SwiGLU (Shazeer, 2020) uses a Swish-gated version:

SwiGLU(x) = (Swish(W_1 * x)) elementwise-times (W_3 * x),  then project down with W_2

The gate lets the network modulate its own activations multiplicatively, which is more expressive than a fixed pointwise non-linearity. Because SwiGLU adds a third weight matrix, implementations shrink the hidden dimension (often to about 8/3 * d instead of 4 * d) to keep the parameter count roughly matched. Llama, Mistral, Qwen, and PaLM all use SwiGLU; the paper's dry conclusion, that it works and it offers "no explanation" for why beyond empirical gain, has become a running joke in the field.

When it falls down

• It is the quantisation and memory bottleneck. Being the largest weights, FFN matrices dominate memory and are the prime target for quantisation and for mixture-of-experts, which replaces one big FFN with many smaller ones activated sparsely (see mixture-of-experts-architecture).
• Gated variants cost a matrix. SwiGLU's third projection means more memory traffic; the hidden-dimension trim keeps parameters level but the extra tensor still has an implementation cost.
• Knowledge is entangled. Facts stored in FFN units are superposed, so editing one can disturb others; clean, surgical model editing remains unreliable precisely because the FFN packs many features into shared weights.

Further reading

• Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762 - the original position-wise FFN.
• Hendrycks and Gimpel, 2016, Gaussian Error Linear Units, arXiv:1606.08415 - GELU.
• Shazeer, 2020, GLU Variants Improve Transformer, arXiv:2002.05202 - SwiGLU and the gated family.
• Geva et al., 2020, Transformer Feed-Forward Layers Are Key-Value Memories, arXiv:2012.14913 - the memory interpretation.