Counting Transformer Parameters

Model cards report parameter counts like "7B" or "70B" as if they were measured rather than computed. They are computed, exactly, from a small set of architectural hyperparameters: layer count, hidden width, feed-forward width, vocabulary size, and number of attention heads. Every weight in a transformer lives in one of a handful of named matrices, and adding up their sizes gives the total to within a rounding error, before you have downloaded a single checkpoint file.

The per-layer budget

Each transformer block contains two weight-bearing sublayers. The attention sublayer has four projection matrices, query, key, value, and output, each roughly d_model x d_model (grouped-query or multi-query attention shrinks the key and value projections; see attention-mechanism), for about 4 * d_model^2 parameters. The feed-forward sublayer expands to an intermediate width d_ff, typically 4 * d_model for a standard MLP or roughly 2.67 * d_model for a gated SwiGLU variant tuned to match parameter count (see feedforward-swiglu), and projects back down, for about 2 * d_model * d_ff parameters (three matrices for SwiGLU's gate, up, and down projections). Normalisation layers add a negligible O(d_model) each. Summed:

params_per_layer ≈ 4 * d_model^2 + 2 * d_model * d_ff
                  ≈ 12 * d_model^2          (when d_ff = 4 * d_model)

The whole model

Multiply the per-layer cost by the number of layers L, then add the parameters that appear exactly once regardless of depth: the token embedding table (vocab_size * d_model) and, if untied, a separate output projection of the same size.

total ≈ L * 12 * d_model^2 + 2 * vocab_size * d_model

For most large models the embedding term is a small fraction of the total, since d_model and L both grow with model size while vocab_size stays roughly fixed (tens of thousands of tokens). This is the basis for the widely used shorthand that total parameters scale approximately as 12 * L * d_model^2, the same quantity Kaplan et al., 2020 used as the backbone of their compute-optimal scaling analysis, and it is why doubling d_model roughly quadruples parameter count while doubling L merely doubles it.

Attention parameters versus attention compute

A subtlety worth holding onto: the parameter count of attention (~4 * d_model^2, independent of sequence length) and the compute cost of attention (O(seq_len^2 * d_model), independent of parameter count in the naive formula) are governed by different variables entirely. A model can have very few attention parameters relative to its feed-forward parameters while attention still dominates wall-clock cost at long context, because parameter count and FLOP count answer different questions (see matmul-the-core-operation).

When it falls down

• The 2N FLOP rule is an approximation, not attention-aware. Kaplan et al., 2020 noted that the 2N-FLOPs-per-token heuristic ignores the quadratic attention term; it is accurate for typical training context lengths and increasingly wrong as context grows.
• Weight tying changes the count. Sharing the input embedding and output projection matrices (common in smaller models) removes an entire vocab_size * d_model term; forgetting to check whether a model ties weights is a common source of a wrong hand-computed total.
• Mixture-of-experts breaks the "active equals total" assumption. In an MoE model, total parameter count and the parameters actually used per token diverge sharply, so quoting a single number without specifying active versus total is misleading (see mixture-of-experts).
• Embedding-heavy models skew the formula. For models with unusually large vocabularies (heavily multilingual tokenisers, for instance), the embedding term is no longer negligible and the 12 * L * d_model^2 shorthand undercounts the true total.

Further reading

• Kaplan et al., 2020, Scaling Laws for Neural Language Models, arXiv:2001.08361 - derives the parameter and FLOP approximations used throughout this concept.
• Hoffmann et al., 2022, Training Compute-Optimal Large Language Models, arXiv:2203.15556 - refines the parameter-versus-data tradeoff (Chinchilla scaling).