Learned Absolute Positions

GPT-2 keeps a table of shape [1024, d_model], one learned vector per position slot from 0 to 1023, summed into the token embedding before the first block. That number, 1024, is not a soft guideline. Feed the model a 1025th token and there is no row of the table to give it; the model has never seen that slot in training and has nothing principled to do with it. Learned absolute position embeddings are the simplest scheme in this whole family, and that simplicity comes with a wall built directly into the parameter count.

The mechanism

Unlike sinusoidal encoding, which computes a fixed formula, a learned position table is just another embedding layer: an nn.Embedding-style lookup indexed by absolute position, trained jointly with token embeddings and every other parameter via ordinary backpropagation. There is no frequency schedule to choose, no base constant, no rotation angle. The network is free to discover whatever geometry over "position space" turns out to be useful for the training data, unconstrained by any hand-designed prior.

This was the approach in BERT, GPT-2, and early GPT-3, and it remains attractive precisely because it removes design decisions. If the sinusoidal formula's assumptions happen to be wrong for a given task or data distribution, a learned table simply is not bound by them.

The wall

The cost is a hard positional ceiling. The table has exactly max_len rows. Positions beyond that either error outright or force an implementation to reuse, clip, or crudely interpolate rows that were never trained for that role, all of which degrade output sharply rather than gracefully. This is a binary cliff, not a slope: a learned-table model at position max_len - 1 behaves normally, and at max_len it is operating entirely outside its training distribution. Contrast this with ALiBi, whose bias formula is defined and reasonable at any distance, or even sinusoidal encoding, which degrades but does not hard-fail.

Extending a learned-table model's context after the fact means adding new rows and initialising them somehow, then fine-tuning to teach the network what to do with them. This tends to work worse and cost more than the relative-distance extension tricks available to RoPE-based models (see RoPE scaling), which is one of the concrete reasons the field moved away from learned absolute tables in frontier decoder-only models after roughly 2022.

What it costs and what it buys

The parameter overhead is trivial at typical scale, max_len * d_model extra weights is a rounding error against a multi-billion parameter model, so the practical downside is not memory. It is inflexibility: changing the maximum supported length means changing the architecture and retraining, not adjusting an inference-time formula. And because attention has to learn "these two absolute position rows are five apart" indirectly, by comparing two separately-summed vectors, it gets a weaker built-in signal for local, relative structure than schemes that put relative distance directly into the attention score.

When it falls down

Hard ceiling, not graceful decay. There is no row for positions beyond max_len; the model is not "worse" past that point, it is undefined.
No relative bias for free. Two tokens five apart and two tokens five hundred apart both rely on the network inferring distance from two absolute embeddings added at the input; nothing in the architecture hands it that difference directly.
Retrofitting length is expensive. Extending context requires adding and training new table rows, unlike training-free or lightly-tuned extensions available to relative schemes.
Memorisation risk. With enough capacity and data, the table can latch onto spurious position-specific correlations in training data (a fact that only ever appears near slot 12, say) rather than genuine positional structure, and those correlations do not generalise.

The mechanism

The wall

What it costs and what it buys

When it falls down

Further reading