Learn the stack, end-to-end.

Attention Mechanism

How attention lets a model focus on the relevant parts of a sequence by computing weighted dependencies between every pair of positions.

Embeddings and Semantic Search

How dense vectors turn text into a geometry of meaning, and how cosine similarity lets you find related content without keywords.

Tokenisation and BPE

How models turn raw text into integer tokens, and why the vocabulary choice silently shapes model behaviour.

Transformer Architecture

The encoder-decoder stack that replaced recurrence and powered every modern LLM.

Calculus and Gradients

Partial derivatives, the chain rule as the engine of backprop, why second-order methods are rare in deep learning, and what gradient clipping actually does.

Linear Algebra for ML

Vector spaces, matrix decompositions, and why low-rank structure underlies LoRA, PCA, and the quantisation tricks that make modern LLMs cheap to serve.

Numerical Computation Gotchas

Catastrophic cancellation, the log-sum-exp trick, mixed-precision training, the determinism tax, and how to actually debug a NaN in a 70B model.

Optimisation Theory

Convexity, why SGD finds good solutions on non-convex losses, saddle points at scale, momentum as a damped oscillator, and learning-rate schedules as implicit regularisation.

Probability and Information Theory

Distributions, expectations, entropy, KL, and why softmax + cross-entropy is the canonical pair that secretly underlies almost every LLM loss.

Statistical Learning Theory Primer

Bias-variance, PAC-learning, VC dimension, why deep nets break classical generalisation bounds, double descent, and what scaling laws are actually saying.

Backpropagation and Automatic Differentiation

How reverse-mode autodiff turns the chain rule into an efficient gradient algorithm, and the design choices PyTorch and JAX make to implement it at scale.

Convolutional Neural Networks

Why weight sharing and local receptive fields make CNNs the right inductive bias for images, and where ViTs took over.

Dropout and Modern Regularisation

Why dropout was the dominant regulariser for a decade and why modern LLM training mostly skips it in favour of letting data do the work.

Normalisation: BatchNorm, LayerNorm, RMSNorm

Why normalisation accelerates training, why transformers use LayerNorm instead of BatchNorm, and why RMSNorm is now the default in Llama-class models.

Optimisers: SGD, Adam, AdamW, Lion

How the standard optimiser stack evolved from plain SGD through Adam to memory-cheaper variants like Lion and Muon, and which learning-rate schedules actually work at scale.

Recurrent Networks: RNN, LSTM, GRU

How gating fixed the vanishing-gradient problem in RNNs, and why transformers displaced them everywhere except streaming and on-device workloads.

Chain of Thought Prompting

Why telling the model to think step by step radically improves reasoning, and when it actively hurts.

Fine-tuning vs RAG

When to teach the model new behaviour vs when to retrieve fresh context at runtime.

Mixture of Experts

Why MoE models can be 10x cheaper to serve than dense models of the same capability, and what makes them hard to train.

Reinforcement Learning from Human Feedback

How preference data and PPO turn a pretrained language model into a helpful, honest, harmless assistant.

Retrieval Augmented Generation

The end-to-end RAG pipeline from chunking through retrieval, reranking, and grounded generation.

FlashAttention

An IO-aware attention kernel that is both faster and lower-memory than the textbook implementation by tiling computation to keep activations in SRAM.

KV Cache

Why decoder inference is quadratic without a KV cache and linear with one, and why managing that cache is now the dominant memory problem in LLM serving.

Mixture-of-Experts Inference

Why serving MoE models is harder than serving dense models of equivalent quality, and how DeepSeek and Mistral made it work in production.

Quantisation - INT8, INT4, FP8

How to cut weight and activation precision below 16 bits without wrecking quality, and which scheme to pick for which deployment.

Speculative Decoding

Use a small draft model to propose tokens that a large verifier accepts or rejects in parallel, giving lossless 2-3x latency wins on autoregressive generation.

vLLM and Continuous Batching

Why static batching wastes most of your GPU on variable-length workloads, and how iteration-level scheduling combined with PagedAttention raises throughput by an order of magnitude.

Data Parallelism and DDP

How replicating the model and sharding the batch across GPUs scales training, and why AllReduce is the primitive every framework eventually depends on.

Gradient Checkpointing, Activation Recomputation, and CPU Offload

Why activations - not weights - usually dominate training memory, and how recomputation and CPU/NVMe offload trade compute and bandwidth to fit larger models.

Mixed-Precision Training (FP16, BF16, FP8)

How lower-precision formats halve memory and double throughput on tensor cores, why BF16 displaced FP16 for training, and what FP8 changes on H100 and Blackwell.

Tensor and Pipeline Parallelism

How frontier labs split a model across thousands of GPUs by sharding within layers (tensor parallel) and across layers (pipeline parallel), and how to pick the split.

ZeRO and FSDP

How sharding optimiser state, gradients, and parameters across data-parallel ranks turns a memory problem into a bandwidth problem, and why FSDP is now the PyTorch default.

Chain of Thought, Self-consistency, and Search at Inference

A tour of the inference-time reasoning toolkit - from zero-shot CoT prompts to MCTS-decoded reasoning trees, and when each pays for itself.

DeepSeek-R1 and the Open Reasoning Recipe

How DeepSeek's R1 pipeline produced o1-class reasoning with an open paper, an open model, and a recipe other labs could replicate within weeks.

OpenAI o1, o3 and the Reasoning-Model Family

What is publicly known and what is speculated about OpenAI's reasoning line, why the chain of thought is hidden, and what o3's ARC-AGI result actually proved.

Process Reward Models and Verifiable Rewards

Why scoring every step of a reasoning trace beats scoring only the final answer, and how Ai2 and DeepSeek replaced PRMs entirely with programmatic correctness checks.

Reasoning Evals and the Contamination Problem

A guided tour of the reasoning benchmark canon, why each saturated faster than the field expected, and the move to live held-out evals as the contamination crisis bites.

Test-time Compute Scaling

Why "thinking longer" at inference can substitute for "training bigger", how the trade-off is operationalised as a token budget, and where the strategy stops paying.

Agentic AI and ReAct

From single tool calls to multi-step agents that plan, act, observe, and recover from errors.

Tool Use and Function Calling

How models invoke external tools to fetch data, run code, and take actions in the world.

Custom Evals and LLM-as-Judge

Why every production team eventually builds its own eval set, and how to use LLM judges without being fooled by their well-documented biases.

HELM and Holistic Evaluation

Why a single accuracy number is gameable, and how Stanford's HELM, BIG-bench, and lm-evaluation-harness push evaluation toward a multi-axis picture.

Model Registry, Lineage, and Reproducibility

The infrastructure that answers "which dataset, code, and hyperparameters produced this checkpoint?" - and why you only miss it the first time you cannot reproduce a model.

Production Monitoring and Drift Detection

How to catch silent regressions in deployed LLMs by monitoring input drift, output quality, and per-user randomised experiments before users tell you something is broken.

Public Benchmarks - MMLU, GPQA, HumanEval, MATH

A tour of the academic benchmarks that anchor frontier model launches, and why most of them are saturating, contaminated, or both.

Red-Teaming and Adversarial Evaluation

Why benign benchmark scores do not predict how a deployed model behaves under attack, and the human and automated methods used to find the failures first.

Alignment Evaluations and Frontier-Model Risk

How frontier labs and governments measure dangerous capabilities, what an eval-gated release looks like, and where the regulatory regime sits in 2026.

Constitutional AI and RLAIF

How Anthropic replaced human harmlessness labels with a written constitution and a critique-and-revise loop, and why this makes alignment auditable.

Jailbreaks and Refusal Robustness

How attackers reliably bypass model refusal training, why post-hoc filters are necessary but never sufficient, and how AILuminate measures what remains.

Mechanistic Interpretability Primer

How sparse autoencoders extract human-interpretable features from model activations, what circuit-level analysis buys you for safety, and where the science is still contested.

Prompt Injection

Why LLMs cannot reliably tell instructions from data, how indirect injection weaponises retrieved content, and which partial defences are worth deploying.

Sycophancy, Deception, and Reward Hacking

Why preference-trained models learn to please rather than to be right, what alignment faking is, and why evaluating during training can mislead you.

Hybrid Retrieval - BM25 + Vector + Reranking

Why pure vector search misses exact-match queries, how RRF combines lexical and semantic results, and where a cross-encoder reranker buys back the precision you lost.

LLM Gateways and Routing

Why every serious LLM deployment ends up behind a gateway, and how to choose between LiteLLM, Portkey, OpenRouter, and rolling your own.

Multi-Tenant Serving and Isolation

Serving many tenants from one model is cheap and easy; giving each tenant their own fine-tune is expensive and hard. S-LoRA and per-request LoRA serving collapse the trade-off, but only for tenants who can share a base model.

Prompt Caching Infrastructure

How Anthropic, OpenAI, and vLLM let you reuse the KV cache of repeated prefixes, what the cache key actually is, and the patterns that turn cache hit rate into a real bill reduction.

Token Accounting, Billing, and Quotas

Why a single token counter is not enough, how to attribute spend across users and features without losing your mind, and the patterns that prevent one bad actor from spending the whole month's budget on a Tuesday afternoon.

Vector Databases Compared - pgvector, Qdrant, Milvus, Weaviate, LanceDB

A practitioner's guide to picking a vector store, weighing index trade-offs against the operational cost of running yet another database alongside your primary store.