Learning-Rate Schedules

A learning rate is not one number, it is a curve. Every serious LLM pretraining run ramps the learning rate up, holds or decays it, and brings it down toward a small floor by the end, and the exact shape of that curve is one of the cheapest levers in the whole training stack: it costs nothing extra in compute, and getting it wrong can cost several points of final loss. The optimiser (see optimisers-adam-adamw-lion) decides how to use a gradient; the schedule decides how far to trust it at each point in training.

The standard shape

Almost every modern pretraining recipe uses the same three-part curve: a short linear warmup from near zero up to a peak learning rate (see warmup-and-why-it-helps), followed by a decay down to some small fraction of the peak, usually around 10%, by the final step. The dominant decay shape is cosine annealing, popularised for deep learning by Loshchilov and Hutter's SGDR paper:

lr(t) = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * t / T))

t is the current step, T is the total number of steps the schedule is designed for. The curve starts flat near the peak, falls steeply through the middle of training, and flattens again near the floor. That flat-steep-flat shape is not decorative: it spends most steps either near the peak (making fast progress) or near the floor (making fine, low-noise adjustments), and comparatively few steps in the less useful middle ground.

SGDR's original proposal included periodic warm restarts, snapping the learning rate back up to the peak and re-running the decay. Pretraining almost never does this. A single cosine cycle from warmup to a chosen end step is the default; restarts are mostly a vision-era technique that did not carry over.

Why decay at all

A fixed, never-decaying learning rate keeps injecting the same amount of gradient noise into the parameters no matter how close training is to a good solution. Early in training, the loss surface has large, informative gradients and a big step size is efficient. Late in training, per-example gradients disagree more with each other relative to their average (the signal-to-noise ratio drops, the same phenomenon critical-batch-size is built on), so the same step size mostly moves the parameters around inside a noise ball instead of toward a better point. Decaying the learning rate shrinks that noise ball over time, letting the model settle rather than churn.

The commitment problem

Cosine decay has an awkward property: the formula needs T, the total step count, before training starts. Choose T correctly and the curve lands exactly where you want by the last step. Choose it wrong, and either the run stops early while the learning rate is still relatively high (the model never gets to fully settle), or the run continues past T with the learning rate pinned near its floor for however many extra steps you tack on, which trains at a rate too small to be worth the compute. Hoffmann et al.'s Chinchilla paper documents this directly: models whose cosine schedule was tuned for a token count larger than what they were actually trained on come out measurably under-optimised relative to a schedule matched to the true horizon. Getting the total-step estimate right, before the expensive part of training even starts, is a real planning constraint, and it is the specific problem warmup-stable-decay schedules were designed to remove.

When it falls down

• The horizon has to be known in advance. Standard cosine decay assumes you know your total token budget before you commit compute. Changing your mind mid-run about how long to train means either a suboptimal schedule or restarting the decay.
• One schedule for every parameter. Nearly all recipes apply the same global curve to every layer, even though different parts of a network (embeddings, attention projections, output head) can have different natural update scales. Per-group schedules are rarely tuned in practice, mostly because cosine is already good enough to make the extra complexity not worth it.
• Peak learning rate and schedule shape are coupled. A peak tuned for one decay shape does not transfer cleanly to another; changing from cosine to a different curve without re-tuning the peak is a common source of quietly worse runs.
• The floor value is arbitrary. Decaying to exactly zero versus 10% of peak versus 1% of peak all show up in different papers with little shared justification; it is one of the more folklore-driven knobs in the whole recipe.

Further reading

• Loshchilov and Hutter, 2016, SGDR: Stochastic Gradient Descent with Warm Restarts, arXiv:1608.03983 - the paper that popularised cosine annealing.
• Hoffmann et al., 2022, Training Compute-Optimal Large Language Models, arXiv:2203.15556 - shows schedule-horizon mismatch costs real loss.