Input Embeddings and the Lookup Table

A tokeniser (see tokenisation-bpe) turns text into a sequence of integers, token ids. Those integers have no arithmetic meaning; id 4021 is not "more" than id 17. The very first thing a transformer does is convert each id into a dense vector it can actually compute with, and it does this the simplest way possible: a lookup table.

The mechanism

The embedding matrix E has shape [vocab_size, d_model]. Row i is the learned vector for token id i. Looking up a token is literally indexing: embedding = E[token_id]. There is no multiplication, no nonlinearity, nothing that resembles a "layer" in the usual sense. The entire operation is a gather.

This has a consequence for training that surprises people coming from dense layers: because each forward pass only touches the rows for tokens that actually appear in the batch, gradients only update those rows. A rare token that shows up twice in a training run gets two gradient updates to its row across the whole run; a common token like "the" gets millions. The embedding table trains extremely unevenly by design, and that unevenness is one root cause of the anisotropy discussed in embedding-space-geometry-anisotropy.

Where the vector goes next

The raw lookup is only the start. The original transformer paper (Vaswani et al., 2017) multiplies the embedding by sqrt(d_model) before adding positional information, a scaling choice made so the embedding's magnitude is comparable to the positional signal that gets added on top (see positional-encodings). After that addition, the vector enters the stack of transformer blocks and, through attention and feed-forward layers, becomes progressively context-dependent, the shift covered in static-vs-contextual-embeddings.

It also matters how big this matrix is. A 50,000-token vocabulary with d_model = 4096 is 200 million parameters, purely for the lookup table, before a single attention head exists. For small models this can be a substantial fraction of total parameters, which is one motivation for reusing this same matrix at the output (see weight-tying-embeddings).

Geometry is learned, not designed

Nobody hand-assigns which direction in the d_model-dimensional space means "royalty" or "past tense". The rows of E start as random noise and only acquire structure because training pushes tokens that behave similarly (appear in similar contexts, need similar next-token predictions) toward similar vectors. This is the same distributional pressure that produced word2vec and GloVe's geometry, decades before transformers (see word2vec-glove-history), just now trained end-to-end alongside everything else instead of as a standalone objective.

When it falls down

• Rare and unseen tokens get poorly shaped vectors. A token seen a handful of times in training has a row that barely moved from initialisation; its geometry is close to noise, which is why rare-word and long-tail behaviour is unreliable in ways common-word behaviour is not.
• The table can dominate model size at small scale. For a 100M-parameter model with a 50k vocabulary and d_model = 1024, the embedding table alone is over 50 million parameters, roughly half the model, spent before any reasoning capacity exists.
• A lookup has no built-in notion of similarity. There is nothing in the mechanism itself that makes "cat" and "dog" end up near each other; that only happens because training makes it useful, and it is not guaranteed for every pair of related tokens.
• The vector at this stage carries no context. "Bank" gets exactly one starting row regardless of whether the sentence is about rivers or money; context only enters through subsequent layers, not the lookup itself.

Further reading

• Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762 - introduces the embedding scaling and shared embedding/softmax weights used in the original transformer.
• Mikolov et al., 2013, Efficient Estimation of Word Representations in Vector Space, arXiv:1301.3781 - the earlier proof that a trained lookup table alone can encode meaningful geometry.