Gradient Clipping

Open the training config of GPT-3, PaLM, Llama, or almost any published LLM recipe and you will find a gradient clip threshold, almost always set to 1.0. It is one of the least discussed hyperparameters precisely because it rarely needs tuning, and one of the most misunderstood, because "we clip gradients" gets treated as a stronger guarantee against instability than it actually provides.

Global-norm clipping, the default

The standard form rescales the entire gradient vector when its overall magnitude exceeds a threshold, preserving direction (see the mechanics in calculus-and-gradients):

if ||grad||_2 > threshold:
    grad = grad * (threshold / ||grad||_2)

This is a global norm computed across every parameter in the model, not per layer or per tensor. That matters: a single layer with an unusually large gradient can dominate the global norm and trigger clipping for the entire model, scaling down every other layer's update along with the offending one, even if those other layers' gradients were perfectly reasonable. Per-tensor or per-layer clipping avoids this coupling but is rarer in LLM pretraining, where global-norm clipping at threshold 1.0 is close to a universal convention. Brock et al.'s Adaptive Gradient Clipping, developed for normalisation-free vision networks, scales the threshold by each parameter's own weight norm instead of using one global cutoff; the idea generalises but has not displaced global-norm clipping as the LLM default.

What it is actually protecting against

Origin, the argument in Pascanu, Mikolov, and Bengio for recurrent networks: occasionally a batch, or a moment in training, produces a gradient dramatically larger than typical, and taking the full, unclipped step in that direction can throw the parameters somewhere the optimiser then has to spend a long time recovering from, if it recovers at all. Clipping caps how far a single bad step can push the model. At LLM pretraining scale the usual triggers are a degenerate batch (heavy repetition, corrupted or duplicate-heavy shards), or a transient mismatch between a parameter's history-normalised update size in Adam and this step's actual gradient. Clipping does not diagnose which of these happened, it just caps the damage.

A safety net, not a cure

Two things follow from clipping happening on the raw gradient, before it reaches the optimiser. First, clipping firing on essentially every step is a signal, not a feature: it usually means the learning rate is too high or the initialisation is poorly scaled relative to what the model can tolerate, and the fix is to address that root cause, not to rely on clipping to paper over it indefinitely. In a healthy run, clipping should trigger rarely, mostly early in training, and then settle to almost never. Second, because Adam's moment estimates (m_t, v_t) are built from whatever gradient reaches the optimiser, a gradient that gets clipped frequently is quietly feeding a downward-biased signal into those moving averages, which subtly changes the optimiser's effective behaviour beyond just capping that one step.

When it falls down

• It cannot fix vanishing gradients. Clipping only caps large gradients; there is no rescaling your way out of a gradient that is already numerically zero. That problem is architectural (residuals, normalisation), not a clipping-threshold problem.
• It does not prevent every kind of divergence. Clipping addresses outlier gradient magnitude, not precision issues or systematic learning-rate misconfiguration; large training runs still see loss spikes despite clipping being active throughout (see loss-spikes-and-divergence).
• It interacts with gradient accumulation. Clipping must happen on the fully accumulated gradient across all microbatches, not on each microbatch's partial gradient individually; clipping too early silently distorts the effective global batch gradient.
• The threshold is rarely re-derived. Most teams copy 1.0 from a prior recipe without ablating it for their specific model and data. That is usually fine, since clipping is a rare-event safety net rather than a precision instrument, but it means clipping frequency is a diagnostic signal that goes unlogged and unused far more often than it should.

Further reading

• Pascanu, Mikolov, and Bengio, 2013, On the Difficulty of Training Recurrent Neural Networks, arXiv:1211.5063 - the original gradient-clipping analysis.
• Brock et al., 2021, High-Performance Large-Scale Image Recognition Without Normalization, arXiv:2102.06171 - Adaptive Gradient Clipping, a per-layer alternative to the global norm.