Weight Decay in Pretraining

Weight decay's textbook justification is overfitting control: shrink weights toward zero so a model does not memorise a fixed, finite training set it sees many times. A large LLM pretraining run typically sees each token once, occasionally twice. There is no long tail of repeated exposure for weight decay to guard against in the classical sense, and yet essentially every published recipe, GPT-3, Llama, and most of what came after, sets weight decay to something like 0.1 and keeps it there for the entire run. Keeping a regulariser in a regime where its stated purpose barely applies is worth asking about directly.

The decoupling AdamW fixed

Under Adam, adding an L2 penalty to the loss does not behave the way it does under plain SGD: the penalty term gets divided by sqrt(v_hat) along with the rest of the gradient, so parameters with a large adaptive denominator get proportionally less decay than they should. Loshchilov and Hutter's AdamW paper decouples the two, applying decay directly to the weights, untouched by Adam's per-parameter scaling (see the full derivation in optimisers-adam-adamw-lion). Every parameter shrinks by the same fractional amount regardless of its gradient history. This fix alone closed a real generalisation gap between Adam and SGD, and is why AdamW, not Adam, is the pretraining default.

What decay is actually doing without overfitting to prevent

The more interesting question is why decay still helps at all in a single-pass regime. The emerging framing is that its benefit in pretraining is less about classical regularisation and more about optimisation dynamics. Many transformer weight matrices sit immediately before a normalisation layer (layernorm-residual-connections if available), which makes them scale-invariant: multiplying such a weight matrix by a constant changes its norm but not the function the network computes, because the normalisation layer immediately renormalises it away. Under a fixed learning rate, unconstrained weight norms in a scale-invariant layer tend to grow over training, and as the norm grows, a fixed-size update becomes a smaller and smaller angular step, the direction the weight actually points barely moves even though its magnitude keeps increasing. van Laarhoven's analysis works this out for the norm-and-decay interaction directly: decay counteracts that drift, keeping the effective step size from shrinking on its own as training proceeds. More recent work, including D'Angelo et al.'s "Why Do We Need Weight Decay in Modern Deep Learning?", argues this optimisation-dynamics effect, not variance reduction on a revisited dataset, is the dominant reason decay still helps when data is seen once.

What gets excluded

Most recipes do not apply decay uniformly. One-dimensional parameters, normalisation gains and biases, and often the embedding table, are commonly excluded from decay, since the scale-invariance argument that motivates decay elsewhere does not apply to them the same way, and decaying them can measurably hurt rather than help. This is a widespread convention across published training code rather than a universal rule with a single settled justification.

When it falls down

• It interacts with the learning-rate schedule. A decay value tuned for one peak learning rate is not automatically right for a different peak; the two are tuned as a pair more often than either is tuned independently, and reports that vary only one without the other should be read cautiously.
• Too much decay in a short run visibly costs early progress. Since there is no long-run overfitting problem to trade against in a single-epoch setting, an oversized decay value mostly shows up as slower early loss descent with no compensating benefit later.
• Blanket application can hurt. Applying decay to embeddings or normalisation gains without the usual exclusions is a common source of quietly worse runs, which is why the exclusion convention exists even without a fully settled theoretical account of it.
• The mechanism is still not fully closed science. The angular-step and effective-learning-rate framing is the best current explanation, but treat any single, confidently stated "this is exactly why weight decay works" claim, including this one, as a working hypothesis rather than a proven law.

Further reading

• Loshchilov and Hutter, 2017, Decoupled Weight Decay Regularization, arXiv:1711.05101 - the AdamW paper.
• van Laarhoven, 2017, L2 Regularization versus Batch and Weight Normalization, arXiv:1706.05350 - the norm-growth and effective-learning-rate argument.
• D'Angelo et al., 2023, Why Do We Need Weight Decay in Modern Deep Learning?, arXiv:2310.04415 - the optimisation-dynamics case for decay outside the classical overfitting story.