The NoPE Result

In 2023, Kazemnejad and colleagues trained decoder-only transformers with no positional encoding at all: no sinusoidal input, no learned table, no RoPE rotation, no ALiBi bias, nothing. On a suite of length-generalisation benchmarks for algorithmic reasoning tasks, these NoPE models matched or outperformed comparable models trained with RoPE or ALiBi. This complicated a tidy story in which some explicit positional mechanism is treated as strictly necessary for a transformer to know order at all.

Why this is possible: the causal mask already leaks position

Plain self-attention over an unordered set of tokens is permutation equivariant (see why attention needs positions): permute the input and the output permutes identically, with nothing distinguishing "first" from "last". But a decoder-only model uses a causal mask, restricting query i to attend only to keys j <= i. That mask alone breaks the permutation symmetry, because the number of tokens visible to query i is a strictly increasing function of i. In principle, a model can recover an absolute position signal simply by counting how many keys it is allowed to see, and recover relative distance from patterns over that visible-count structure, with no explicit encoding telling it to.

What the paper found mechanistically

Probing the trained NoPE models, the authors found attention patterns that closely resembled relative positional schemes, learned entirely from the causal-mask scaffold with no explicit signal ever provided. On the length-generalisation tasks tested, this implicitly learned structure generalised out of distribution better than the explicit RoPE and ALiBi models in the comparison. The causal mask, in other words, was already supplying enough of an inductive bias that training discovered something functionally similar to a relative positional scheme on its own.

The scope of the claim, stated carefully

The strongest NoPE results are on small models trained on algorithmic length-generalisation suites, not on the trillion-token, frontier-scale pretraining regime used for production chat models. Whether the same advantage holds at that scale and with natural-language data is genuinely unresolved, and the original paper is careful not to overclaim beyond its tested setting. No widely deployed frontier decoder-only model currently ships without an explicit positional scheme, and RoPE remains the default even in labs aware of this result, largely because it offers controllable, predictable behaviour under the context-extension techniques covered in RoPE scaling: NTK-aware and YaRN; betting a 100k-plus-token production context window on an implicit, uncontrolled signal is a materially different risk profile than relying on an explicit rotation with well-studied scaling knobs.

The conceptual payoff

The useful reframing is not "positional encoding is unnecessary" but "the causal mask is itself a weak, real source of order information, and explicit positional encoding is a strong, controllable prior layered on top of it, not the sole source of order". This distinction has a sharp boundary: bidirectional, non-causal architectures like BERT have no mask-derived signal at all, so nothing in the NoPE result suggests they could drop explicit positional encoding. The finding is specific to causal, autoregressive decoders, exactly the architecture family that dominates current LLMs, which is what makes it interesting rather than a footnote.