Loss Spikes and Divergence

Plot the loss curve of nearly any large, long pretraining run and, sooner or later, the smooth downward line breaks: a sudden jump upward, sometimes recovering within a few hundred steps, sometimes never recovering without intervention. Chowdhery et al.'s PaLM paper is unusually candid about this, reporting roughly twenty such spikes across the full training run despite standard mitigations (warmup, gradient clipping, careful initialisation) being active the entire time. Loss spikes are not a sign of an amateur setup; they are close to a universal feature of training at this scale, and how a team responds to them is one of the least publicised but most consequential parts of running a frontier pretraining job.

What actually causes one

No single mechanism explains every spike, but three recur across published post-mortems.

Degenerate batches. A shard of highly repetitive, corrupted, or otherwise unusual data can produce a gradient that is unusually large and, worse, correlated across many parameters in the same direction. Global-norm gradient clipping rescales the whole vector's magnitude but preserves its direction, so a correlated bad gradient still pushes every affected parameter the same way, just by a smaller amount. Clipping blunts the damage; it does not neutralise a systematically bad update.

Adam's per-parameter scaling going stale. Adam divides each parameter's update by sqrt(v_hat), a running estimate of that parameter's typical squared gradient magnitude. If a parameter has had a quiet stretch (small v_hat) and then receives an unusually large gradient, its individual update can be large even after the gradient vector as a whole was clipped, because clipping applies a single global rescaling factor, not a per-parameter one. A handful of parameters going briefly out of sync with their own recent history is enough to move the whole model measurably.

Precision effects. Training in bf16 gives roughly three decimal digits of mantissa precision. If the softmax normaliser (the log-partition function inside cross-entropy, see softmax-logits) drifts to an unusually large or small value, small numerical errors in that computation can translate into a disproportionately large gradient. PaLM's mitigation was an auxiliary z-loss term that lightly penalises the log of the softmax normaliser for straying from zero, keeping the numerics in a well-behaved range without materially changing what the model learns. This interacts directly with format choice; see mixed-precision-bf16-fp8 for the broader precision picture.

What teams actually do

The pragmatic response documented in the PaLM report is unglamorous: roll the training state back to a checkpoint saved roughly a hundred steps before the spike, skip the batches of data that were being consumed around the time of the spike, and resume from there. This happened repeatedly over the course of the run. It is a workaround, not a fix, and it depends on checkpointing and data-shard bookkeeping being frequent and precise enough to pinpoint what to skip. OPT's public training logbook documents a similarly hands-on process of restarts, hyperparameter adjustments, and hardware failures interleaved with loss instability, offering one of the few fully transparent accounts of how messy this part of training actually is.