The Critical Batch Size

Data parallelism scales by widening the batch: more replicas, more sequences per optimiser step, in principle a proportionally shorter wall-clock training time. In principle, because past a certain batch size that speedup stops being proportional. A team with twice the GPUs does not get a training run that finishes in half the time forever; at some point, doubling the batch barely reduces the number of optimiser steps needed to reach a given loss, and all those extra GPUs are computing largely redundant gradient information. The point where this transition happens is the critical batch size, and it is one of the more load-bearing, least intuitive numbers in large-scale training.

The gradient noise scale

McCandlish et al.'s empirical study of large-batch training frames this through the gradient noise scale: a single minibatch gradient is a noisy estimate of the true gradient over the full data distribution, and the ratio of that noise's variance to the true gradient's squared magnitude sets a natural batch size below which averaging more examples together genuinely sharpens the estimate, and above which it mostly averages out noise that was already small relative to the signal. Roughly:

B_crit ~ trace(gradient_covariance) / ||true_gradient||^2

Below B_crit, doubling the batch size lets you roughly halve the number of steps needed for the same progress, a close-to-linear trade of more parallel compute for less wall-clock time. Above B_crit, doubling the batch keeps buying only marginal step-count reduction; the compute cost of the wider batch stops being repaid in faster convergence.

It moves during training

B_crit is not a fixed property of a model, it is closer to a property of the current loss. Early in training, when the loss is high and gradients carry a lot of signal relative to their noise, B_crit tends to be smaller and grows as training proceeds and the easy, high-signal progress is used up; near convergence, gradients are noisier relative to their magnitude and B_crit grows further, meaning very large batches become comparatively more efficient later in training than early on. Kaplan et al.'s scaling-laws paper reports this same connection between loss level and critical batch size, tying gradient noise scale directly into the compute-optimal training framework.

The linear scaling rule, and where it stops applying

Goyal et al.'s large-minibatch SGD work established the practical corollary: when you increase the batch size by a factor k, scale the learning rate by roughly the same factor k to keep the optimisation dynamics equivalent. This rule holds well below the critical batch size. Push the batch size past B_crit and keep scaling the learning rate with it, and the result is frequently the opposite of stability: a learning rate calibrated for a batch size the gradient noise can no longer support is a well-documented trigger for loss spikes. Choosing a pretraining batch size is, in practice, choosing a point near but not far past B_crit for the compute budget at hand, then using gradient accumulation to hit that target batch size across however many GPUs are actually available.