The Causal LM Pretraining Task

A causal language model is never handed a labelled dataset in the way an image classifier is. It is handed raw text, and the entire supervision signal comes from a single rule applied to every position in every document: predict the next token, using only what came before it. That one rule, applied billions of times, is the whole task. Understanding how it turns a folder of text files into a training run is separate from understanding the loss function itself (see next-token-prediction-cross-entropy); this concept is about the mechanics that get you there.

From document to training example

Pretraining corpora are not stored as (input, label) pairs. A pipeline tokenises each document (see tokenisation-bpe) into a long stream of integer token IDs, then chops that stream into fixed-length chunks matching the model's context length, T. Each chunk of length T becomes one training example, and its label sequence is simply the chunk shifted left by one position:

tokens  = [t1, t2, t3, ..., tT, t(T+1)]
inputs  = [t1, t2, t3, ..., tT]
labels  = [t2, t3, t4, ..., t(T+1)]

There is no separate "target" data anywhere in the corpus; the label at every position is just the token that happens to sit next to it. This is what makes causal LM pretraining cheap to supervise at trillion-token scale: labelling is free, because the text already contains the answer key.

Enforcing "no peeking"

Predicting t2 from t1 only is easy to arrange for a single pair, but a transformer processes the whole chunk of T tokens in one forward pass for efficiency, not one token at a time. Left unconstrained, self-attention would let the token at position 5 look at position 9 and simply copy it, an answer the model would never have at inference time. The fix is the causal (or "look-ahead") mask applied inside every attention layer: position i may attend only to positions 1..i.

score[i][j] = -infinity                if j > i
score[i][j] = QK^T / sqrt(d_k)         if j <= i

Masked positions get -infinity before the softmax, so they receive exactly zero attention weight (see softmax-logits for why that zeroing is exact and not approximate). This single triangular mask is what makes the factorisation p(x) = prod_t p(x_t | x_<t) true of the model's forward pass, not just of the maths on paper, and it is the reason the architecture is called decoder-only or causal, independent of the loss used on top of it (see transformer-architecture).

One forward pass, T predictions

Because the mask already prevents leakage, the model can score every position in the chunk simultaneously: position 1 predicts t2, position 2 predicts t3, and so on, all inside one forward pass over one chunk. This parallel scoring (teacher forcing, covered in detail in next-token-prediction-cross-entropy) is what makes pretraining throughput viable; without the causal mask you would need T separate forward passes to safely predict T tokens, one per prefix length.

Why "causal" is a pipeline decision, not just a formula

It is tempting to think of "causal LM" purely as a loss function, but the mask, the chunking, and the left-shifted labels are three separate engineering choices that all have to agree for the task to be coherent. Change the mask to bidirectional and you get masked-language-modelling instead, a genuinely different task with different data preparation. Change how chunks are assembled from multiple documents and you get the packing problem (see sequence-packing). The loss (cross-entropy) is the same in most of these variants; what differs is what the model is allowed to see when it makes each prediction.

When it falls down

• Arbitrary chunk boundaries. Fixed-length chunking cuts documents at token positions T, 2T, 3T, ... with no respect for sentence or document boundaries, so a chunk can begin mid-sentence with no way for the model to know what preceded it.
• Short documents waste capacity. A document shorter than T either gets padded (wasted compute on pad tokens) or concatenated with unrelated documents (see sequence-packing for the leakage risk this introduces).
• Left-only context is a permanent ceiling. No amount of scale gives a causal model access to future tokens during training or inference; tasks that benefit from seeing the whole input at once (classification, retrieval encoding) are systematically better served by bidirectional objectives (see masked-language-modelling).
• The pretraining task is not the deployment task. A model that has only ever predicted the next token of raw web text has no notion of "answer the user's question and stop"; that behaviour is added later, using the same shifted loss, just selectively masked (see loss-masking-in-fine-tuning).

Further reading

• Radford et al., 2018, Improving Language Understanding by Generative Pre-Training - the paper that established decoder-only causal pretraining followed by fine-tuning as a recipe.
• Vaswani et al., 2017, Attention Is All You Need - introduces the masked self-attention mechanism that makes the causal constraint enforceable inside a single forward pass.