Warmup-Stable-Decay Schedules

Cosine decay's formula needs the total step count T before training begins, lr(t) is a function of t / T. Commit to T, then decide partway through that you want to train longer or shorter, and the neat curve heading smoothly toward a small floor at the planned final step is now wrong for the run you are actually doing. Warmup-Stable-Decay (WSD) schedules, described in detail in Hu et al.'s MiniCPM report, restructure the curve so that the horizon-commitment problem shrinks to a small tail at the end of training instead of being baked into every step.

The three phases

Warmup, same short linear ramp as any other schedule (see warmup-and-why-it-helps). Stable, a long plateau held at a constant peak learning rate, for as many steps as desired; because a constant rate has no notion of a fixed endpoint, this phase can be extended on the fly without invalidating anything that came before. Decay, a comparatively short, steep drop applied only near the very end of training, over a modest fraction of total steps, bringing the learning rate down to its floor quickly rather than gradually across the whole run.

Why the split is useful beyond convenience

Checkpoint reusability. Any checkpoint saved during the stable phase sits at the same constant learning rate as every other point in that phase, so it is a fully valid place to resume, extend, or branch training from. A checkpoint saved partway through a cosine decay, by contrast, has an learning-rate trajectory that already encodes a specific commitment to a specific total length; resuming it cleanly for a different horizon is awkward.

Cheap scaling-style experiments. The MiniCPM report describes branching several independent short decay phases off different points along a single stable-phase run, each one simulating "what if training had stopped here." That produces a family of final-loss estimates at different token counts from one underlying run, rather than requiring several separate full training runs, similar in spirit to the questions scaling-laws-chinchilla style analysis usually needs many independent runs to answer.

Continual and domain-adaptive training. A checkpoint saved mid-stable-phase can keep training on a shifted or newly available data mix without discarding progress the way restarting a cosine schedule already committed to a specific length would.

How it compares to cosine

Reported final loss under WSD is close to, though not consistently better than, a well-tuned cosine schedule of matching total length. The real advantage is operational: not having to commit to the total token budget upfront, rather than a guaranteed loss improvement over cosine. A team that already knows its exact compute budget in advance, and has no need to branch checkpoints or extend training later, has correspondingly less reason to prefer WSD over a standard cosine curve; the choice is about flexibility under uncertainty, not a strictly better optimisation result. The comparison is naturally against learning-rate-schedules's cosine baseline, since WSD is best understood as an alternative to that default rather than an unrelated technique.