In-Context Learning

The defining discovery of the GPT-3 era was not that a bigger model scored higher. It was that a big enough model could learn a new task from a handful of examples placed in its prompt, adjusting no weights at all (Brown et al., 2020, Language Models are Few-Shot Learners, arXiv:2005.14165). Show it three translations and it translates the fourth; show it a labelling pattern and it continues the pattern. The "learning" happens entirely inside a single forward pass.

The three regimes

• Zero-shot: just the instruction. "Translate to French: hello".
• Few-shot: the instruction plus a few worked examples before the real query. The examples demonstrate the format and the mapping.
• Many-shot: dozens or hundreds of examples, viable once context windows grew large enough to hold them.

The examples are not training data in any formal sense. They are conditioning: they steer the model's next-token distribution toward the demonstrated behaviour. Delete them from the prompt and the ability vanishes, because nothing was stored.

Why it is surprising

Ordinary machine learning learns by gradient descent on weights. In-context learning produces task-specific behaviour with a frozen network, which means the adaptation lives in the activations, not the parameters. A useful way to hold it: the weights encode a vast space of possible behaviours, and the prompt selects a point in that space. One influential line of work argues the forward pass can implement something resembling an optimization step internally, effectively "learning" from the examples as it reads them, though the exact mechanism is still debated.

It is an emergent property of scale

Small models barely show in-context learning; the ability strengthens sharply with scale. That scale-dependence is why it went unnoticed for years, and why it reframed LLMs from "text predictors you fine-tune" into "general engines you program with a prompt". It is the direct ancestor of prompt engineering and, once tasks got complex, of context engineering.

The catch

In-context learning is powerful but shallow and brittle. It is sensitive to example order, formatting, and the specific examples chosen; swap two demonstrations and accuracy can move measurably. It also consumes context budget, every example is tokens the model must process on every call, which collides directly with context rot once prompts get long. For durable, high-volume behaviour, fine-tuning is often the better tool; in-context learning wins on flexibility and zero setup.

Further reading

• Brown et al., 2020, Language Models are Few-Shot Learners, arXiv:2005.14165 - the paper that named few-shot in-context learning.