Broadcasting and Vectorisation

Add a (d_model,) bias vector to a (batch, seq_len, d_model) activation tensor and it just works, no loop, no explicit reshape. That convenience has a name, broadcasting, and a precise rule behind it. Broadcasting is also the reason vectorised code (operating on whole tensors at once) is both faster and shorter than the equivalent Python loop over individual elements, which is why virtually no production deep learning code contains an explicit loop over batch or sequence positions.

Vectorisation: why loops are the wrong tool

A Python-level loop over, say, every position in a sequence, calling a function once per position, pays interpreter overhead on every single call and cannot use the hardware's parallel execution units. Vectorisation replaces that loop with a single operation over the entire tensor, letting the underlying compiled kernel (running on CPU SIMD instructions or GPU cores) process every element in parallel. This is not a minor speedup, it is routinely one or two orders of magnitude, and it is the reason deep learning frameworks are built around the discipline of expressing computation as whole-tensor operations rather than per-element Python logic (see matmul-the-core-operation for the matrix-multiply case specifically).

The broadcasting rule

Broadcasting lets operations combine tensors of different shapes by virtually (not physically) expanding the smaller one, without copying data. The rule, applied by comparing shapes from the rightmost dimension leftward:

1. Two dimensions are compatible if they are equal, or if one of them is 1.
2. A missing dimension (one tensor has fewer axes) is treated as size 1 and prepended.
3. Wherever a dimension is 1, that tensor is virtually repeated along that axis to match the other operand's size.

(batch, seq_len, d_model)   activations
+          (d_model,)        bias
-> bias is treated as (1, 1, d_model), then broadcast to (batch, seq_len, d_model)

This is exactly how a single bias vector applies identically to every position of every sequence in a batch, and how a single scaling scalar multiplies an entire tensor, without the framework ever materialising an expanded copy in memory; the expansion is logical, computed on the fly during the operation.

Where it shows up in a transformer

Beyond bias addition, broadcasting is what lets an attention mask of shape (batch, 1, 1, seq_len) apply identically across all heads and all query positions when added to attention scores of shape (batch, heads, seq_len, seq_len), the size-1 axes on the mask expand to match. It is also how a per-example loss weight of shape (batch, 1) can scale a per-token loss tensor of shape (batch, seq_len) without a loop. Recognising the broadcasting pattern is often the fastest way to understand what a one-line masking or weighting operation in unfamiliar code is actually doing.

When it falls down

• Silent shape coincidences. Broadcasting does not check that a size-1 axis dimension makes semantic sense, it only checks that shapes are compatible. Two tensors with unrelated meanings can broadcast together without error if their shapes happen to line up, producing wrong results with no warning (this is the sharpest edge of broadcasting, and worth the extra second of shape-checking before trusting a new line of code).
• Right-to-left alignment surprises newcomers. Broadcasting aligns shapes from the last dimension backward, so a (seq_len,) tensor broadcasts against the last axis of (batch, seq_len, d_model) only if seq_len happens to equal d_model, or an explicit axis needs to be inserted (commonly via unsqueeze or None indexing) to broadcast against the intended axis instead.
• It can mask a missing loop that should exist. Occasionally a bug is written as "broadcasting shaped" (a size-1 axis where a real per-example value was intended), and because broadcasting resolves without error, the mistake produces a plausible-looking but wrong result rather than a crash.
• Memory savings are logical, not universal. While broadcasting itself avoids materialising the expanded tensor, a subsequent operation may still force materialisation (for instance, if the result is stored or passed to a kernel that requires a fully realised tensor), so "broadcasting is free" is a rule of thumb, not a guarantee, for every downstream operation.

Further reading

• NumPy documentation, Broadcasting - the canonical statement of the broadcasting rule, adopted essentially unchanged by PyTorch, JAX, and TensorFlow.
• PyTorch documentation, Broadcasting semantics - the framework-specific edge cases (in-place operations, gradient behaviour) worth knowing before relying on broadcasting in training code.