Label Smoothing

Cross-entropy against a one-hot target has an uncomfortable property: it is minimised only when the model assigns probability exactly 1 to the correct class, which requires the correct logit to be infinitely larger than every other logit (see softmax-logits). No finite set of weights achieves that, so the optimiser keeps pushing logit margins wider forever, chasing a target it can never reach. Szegedy et al., 2015 introduced label smoothing as a way to remove that impossible target.

The mechanism

Instead of training against a one-hot vector, label smoothing blends it with a uniform distribution over all classes, controlled by a small constant epsilon (commonly 0.1):

y_smooth_i = (1 - epsilon) * y_onehot_i + epsilon / V

where V is the number of classes (the vocabulary size, for a language model's output head). The true class now has a target of 1 - epsilon + epsilon/V instead of 1, and every other class has a small positive target of epsilon/V instead of 0. Cross-entropy against this smoothed target is minimised at a finite logit margin, one where the correct class's probability approaches 1 - epsilon, not 1. The impossible target is gone, and with it the unbounded pressure to keep enlarging logits.

What this buys you

• Calibration. A model trained on hard one-hot targets tends to become overconfident: its top-1 probability is systematically higher than its actual accuracy would justify. Capping the achievable target probability directly counteracts this.
• A cheap regulariser on logit magnitude. Because the loss no longer rewards ever-larger correct-class margins, label smoothing indirectly keeps logits from drifting to extreme values, complementary to explicit magnitude penalties like z-loss.
• A small but real generalisation gain on many classification benchmarks, attributed to the model spending less capacity memorising training-set margins that do not transfer.

Where it stops helping

Muller, Kornblith and Hinton, 2019 ran a systematic study of when label smoothing helps and found a genuine cost: it erases exactly the information that knowledge distillation depends on. A teacher trained with hard labels produces "dark knowledge", the relative probabilities it assigns to wrong classes (a photo of a wolf gets a little probability mass on "husky" and almost none on "car"), and a student model learns useful structure from those ratios. Label smoothing on the teacher flattens wrong-class probabilities toward the uniform epsilon/V floor, destroying the relative ordering among wrong classes that distillation relies on. Their recommendation: smooth if you are training a final model, but do not smooth a teacher you intend to distil.

Whether it belongs in LLM pretraining

Most large-scale causal LM pretraining today uses plain cross-entropy, not label smoothing, on the next-token objective. Two reasons: at trillion-token scale with a single epoch or less over most data, overfitting to specific training examples (label smoothing's original motivation) is a much smaller risk than in the many-epoch image classification regime it was designed for; and the effective vocabulary competition per position already provides a soft target of sorts, since natural language genuinely has multiple plausible next tokens, unlike a hard image label. Label smoothing remains more common in machine translation, smaller-scale classification fine-tunes, and settings closer to its original regime.

When it falls down

• Destroys dark knowledge for distillation, as above; never smooth a distillation teacher.
• Epsilon is a real hyperparameter, not a free lunch. Too large and the model under-fits the true label, visibly capping accuracy; the common default of 0.1 is a starting point, not a universal constant.
• Interacts oddly with very large output vocabularies. With V in the hundreds of thousands (an LLM's token vocabulary), epsilon/V is minuscule per wrong class but the aggregate mass, epsilon, is still fully subtracted from the correct class, so the effective smoothing strength should be re-tuned rather than reused from a 1000-class image setting.
• Does not fix miscalibration caused by other sources, such as distribution shift between training and deployment data; it only addresses the specific overconfidence induced by hard-target cross-entropy.

Further reading

• Szegedy et al., 2015, Rethinking the Inception Architecture for Computer Vision - introduces label smoothing as a regulariser.
• Muller, Kornblith and Hinton, 2019, When Does Label Smoothing Help? - the calibration benefit and the distillation cost.